CN114490272A

CN114490272A - Data processing method and device, electronic equipment and computer readable storage medium

Info

Publication number: CN114490272A
Application number: CN202210143858.8A
Authority: CN
Inventors: 茅逸斐; 吴都; 熊慧君; 徐修颖
Original assignee: Industrial and Commercial Bank of China Ltd ICBC
Current assignee: Industrial and Commercial Bank of China Ltd ICBC
Priority date: 2022-02-16
Filing date: 2022-02-16
Publication date: 2022-05-13

Abstract

The disclosure provides a data processing method and device, electronic equipment and a computer readable storage medium, which can be applied to the fields of operation and maintenance and finance. The data processing method comprises the following steps: sending an acquisition request in response to detecting that information related to a first application program in an application program list is updated, wherein the application program list is used for recording information of at least one application program running on a server, and the acquisition request is used for acquiring a first acquisition tool corresponding to the first application program; in response to receiving the first collection tool, collecting, with the first collection tool, first target data for the first application, wherein the first target data includes and at least one type information; and sending the first target data to the target database according to the at least one type information, so that the target database stores the first target data in a classified manner.

Description

Data processing method and device, electronic equipment and computer readable storage medium

Technical Field

The present disclosure relates to the field of operation and maintenance and the field of finance, and more particularly, to a data processing method and apparatus, an electronic device, a computer-readable storage medium, and a computer program product.

Background

With the convergence of the service system, more and more application programs are run on each server in the service system, which not only increases the working strength of operation and maintenance, but also makes the centralized service system become more complicated.

In the field of operation and maintenance monitoring, attention needs to be paid to application programs running on various servers, and various operation and maintenance systems can be used for monitoring the running conditions of the application programs. In the related art, a plurality of acquisition clients are generally deployed for acquiring data of different applications, and the acquisition is realized by acquiring related information such as logs, indexes, configurations and the like of the applications.

In implementing the disclosed concept, the inventors found that there are at least the following problems in the related art: the acquisition client is usually deployed manually, and when an application program running on the server is changed, the acquisition client needs to be updated manually, so that the operation and maintenance difficulty is high, and the operation and maintenance cost is high; changes of the application program may not be found in time, resulting in data missing or repeated collection; a better processing mode is not available for the acquired data.

Disclosure of Invention

In view of the above, the present disclosure provides a data processing method and apparatus, an electronic device, a computer-readable storage medium, and a computer program product.

According to a first aspect of the present disclosure, there is provided a data processing method including:

sending an acquisition request in response to detecting that information related to a first application program in an application program list is updated, wherein the application program list is used for recording information of at least one application program running on a server, and the acquisition request is used for acquiring a first acquisition tool corresponding to the first application program;

collecting first target data for the first application program by using the first collecting tool in response to receiving the first collecting tool, wherein the first target data comprises at least one type information; and

and sending the first target data to a target database according to the at least one type information so that the target database stores the first target data in a classified manner.

According to an embodiment of the present disclosure, wherein the type information includes at least one of: log data, timing index data and topology association data;

wherein the first target data further includes tag information, the tag information including at least one of: identification information of the server, acquisition time, and version information of the first acquisition tool.

According to an embodiment of the present disclosure, the sending the first target data to a target database according to the at least one type information, so that the target database stores the first target data in a classified manner includes:

transmitting the log original text data and the identification information to a log database when the first target data comprises the log data;

transmitting the time series index data and the identification information to a time series index database when the first target data includes the time series index data;

and transmitting the topology related data and the identification information to a topology related database when the first target data comprises the topology related data.

According to an embodiment of the present disclosure, the method further includes, before the sending of the acquisition request in response to detecting that the information related to the first application in the application list is updated:

scanning the application program running on the server;

determining information of a second application program which is not currently running;

deleting the second application program in the application program list; and

and unloading a second acquisition tool corresponding to the second application program.

According to an embodiment of the present disclosure, the method further includes, before the collecting, with the first collection tool, first target data for the first application in response to receiving the first collection tool:

receiving an installation package or a download link of the first acquisition tool from a cloud platform, wherein the cloud platform comprises a plurality of acquisition tools;

in response to receiving the download link, generating a random parameter according to a storage capacity required when executing the download link;

and acquiring the first acquisition tool corresponding to the download link in a time-sharing manner according to the random parameter.

According to an embodiment of the present disclosure, the method further includes:

and sending the heartbeat packet to a management terminal so that the management terminal generates abnormal alarm information under the condition that the management terminal does not receive the heartbeat packet again in a preset time period.

According to a second aspect of the present disclosure, there is provided a data processing apparatus comprising: the device comprises a first sending module, an acquisition module and a second sending module.

The server comprises a first sending module, a second sending module and a third sending module, wherein the first sending module is used for sending an acquisition request in response to the detection that the information of a first application program in an application program list is updated, the application program list is used for recording the information of at least one application program running on the server, and the acquisition request is used for acquiring a first acquisition tool corresponding to the first application program;

a collection module, configured to collect, in response to receiving the first collection tool, first target data for the first application using the first collection tool, where the first target data includes at least one type information; and

and the second sending module is used for sending the first target data to a target database according to the at least one type information so that the target database stores the first target data in a classified manner.

According to a third aspect of the present disclosure, there is provided an electronic device including:

one or more processors;

a memory to store one or more instructions that,

wherein the one or more instructions, when executed by the one or more processors, cause the one or more processors to implement the method as described above.

According to a fourth aspect of the present disclosure, there is provided a computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to implement the method as described above.

According to a fifth aspect of the present disclosure, there is provided a computer program product comprising computer executable instructions for implementing the method as described above when executed.

According to the embodiment of the disclosure, when it is detected that the related information in the application program list is updated, an acquisition request is sent, the target data is acquired according to the received acquisition tool, and the data is sent to the corresponding database to be stored according to the type information of the target data. Through the technical means, the technical problems that operation and maintenance difficulty is high, data are easy to miss or repeatedly collect, and a good processing mode is not available for collected data in the related technology are at least partially solved, so that the real-time performance of application program data collection is improved, and the diversified requirements of storage and using modes of collected data are met.

Drawings

The above and other objects, features and advantages of the present disclosure will become more apparent from the following description of embodiments of the present disclosure with reference to the accompanying drawings, in which:

FIG. 1 schematically shows a system architecture diagram to which a data processing method may be applied, according to an embodiment of the present disclosure;

FIG. 2 schematically shows a flow chart of a data processing method according to an embodiment of the present disclosure;

FIG. 3 schematically shows an application scenario diagram of a data processing method according to an embodiment of the present disclosure;

FIG. 4 schematically illustrates a flow chart of a method of unloading a second capture tool in accordance with an embodiment of the present disclosure;

FIG. 5 schematically illustrates a flow chart of a method of acquiring a first capture tool in accordance with an embodiment of the present disclosure;

fig. 6 schematically shows an application scenario diagram of a data processing method according to another embodiment of the present disclosure;

FIG. 7 schematically shows a block diagram of a data processing apparatus according to an embodiment of the present disclosure; and

fig. 8 schematically shows a block diagram of an electronic device adapted to implement a data processing method according to an embodiment of the present disclosure.

Detailed Description

Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be understood that the description is illustrative only and is not intended to limit the scope of the present disclosure. In the following detailed description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the disclosure. It may be evident, however, that one or more embodiments may be practiced without these specific details. Moreover, in the following description, descriptions of well-known structures and techniques are omitted so as to not unnecessarily obscure the concepts of the present disclosure.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The terms "comprises," "comprising," and the like, as used herein, specify the presence of stated features, steps, operations, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, or components.

All terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art unless otherwise defined. It is noted that the terms used herein should be interpreted as having a meaning that is consistent with the context of this specification and should not be interpreted in an idealized or overly formal sense.

Where a convention analogous to "at least one of A, B and C, etc." is used, in general such a construction is intended in the sense one having skill in the art would understand the convention (e.g., "a system having at least one of A, B and C" would include but not be limited to systems that have a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.). Where a convention analogous to "A, B or at least one of C, etc." is used, in general such a construction is intended in the sense one having skill in the art would understand the convention (e.g., "a system having at least one of A, B or C" would include but not be limited to systems that have a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.).

In the technical scheme of the disclosure, the acquisition, storage, application and the like of the personal information of the related user all accord with the regulations of related laws and regulations, necessary security measures are taken, and the customs of the public order is not violated.

In the technical scheme of the disclosure, before the personal information of the user is acquired or collected, the authorization or the consent of the user is acquired.

With the development of internet and computer technologies, more and more users are considering or adopting a scheme of service centralization, however, after the service system is centralized, more and more applications are run on each server in the service system, which not only increases the working strength of operation and maintenance, but also makes the centralized system become more complicated.

In the field of operation and maintenance monitoring, attention needs to be paid to application programs running on various servers, and in order to monitor running conditions of the application programs, various operation and maintenance systems have been developed, and in related technologies, multiple acquisition clients are generally deployed for acquiring data of different application programs and are implemented by acquiring related information such as logs, indexes, configurations and the like of the application programs.

In order to at least partially solve the technical problems in the related art, the present disclosure provides a data processing method and apparatus, an electronic device, and a computer-readable storage medium, which may be applied to the operation and maintenance field and the financial field. The data processing method comprises the following steps: sending an acquisition request in response to detecting that information related to a first application program in an application program list is updated, wherein the application program list is used for recording information of at least one application program running on a server, and the acquisition request is used for acquiring a first acquisition tool corresponding to the first application program; in response to receiving the first collection tool, collecting, with the first collection tool, first target data for the first application, wherein the first target data includes and at least one type information; and sending the first target data to the target database according to the at least one type information, so that the target database stores the first target data in a classified manner.

It should be noted that the method and apparatus for data processing provided by the embodiments of the present disclosure may be used in the operation and maintenance field and the financial field, for example, operation and maintenance monitoring of a banking outlet business system. The method and the device for data processing provided by the embodiment of the disclosure can also be used in any fields except the operation and maintenance field and the financial field, such as the management of the server by the management terminal cluster. The application fields of the data processing method and device provided by the embodiment of the disclosure are not limited.

Fig. 1 schematically shows a system architecture to which a data processing method according to an embodiment of the present disclosure may be applied. It should be noted that fig. 1 is only an example of a system architecture to which the embodiments of the present disclosure may be applied to help those skilled in the art understand the technical content of the present disclosure, and does not mean that the embodiments of the present disclosure may not be applied to other devices, systems, environments or scenarios.

As shown in fig. 1, the system architecture 100 according to this embodiment may include

terminal devices

101, 102, 103, a server 104, and a cloud platform 105. The

terminal devices

101, 102, 103 and the server 104, the cloud platform 105 and the server 104 may communicate with each other via a network, which may include various connection types, such as wired and/or wireless communication links, and the like.

The user may use the

terminal devices

101, 102, 103 to interact with the server 104 to receive or send messages or the like. Various applications may be installed on the

terminal devices

101, 102, 103, such as a shopping-like application, a web browser application, a search-like application, an instant messaging tool, a mailbox client, and/or social platform software, etc. (by way of example only).

The

terminal devices

101, 102, 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like.

The server 104 may be a server providing various services, such as a background management server (for example only) providing support for applications downloaded by users using the

terminal devices

101, 102, 103. The background management server may analyze and perform other processing on the received data such as the user request, and feed back a processing result (for example, a download link, information, or data obtained or generated according to the user request) to the terminal device.

The cloud platform 105 may include a plurality of acquisition tools that may be used to download applications. After receiving the acquisition request of the acquisition tool sent by the server 104, the cloud platform 105 may send an installation package or a download link of the corresponding acquisition tool to the server 104.

It should be noted that the method for processing data provided by the embodiment of the present disclosure may be generally executed by the server 104. Accordingly, the data processing apparatus provided by the embodiments of the present disclosure may be generally disposed in the server 104. The method for processing data provided by the embodiment of the present disclosure may also be performed by a server or a server cluster different from the server 104 and capable of communicating with the

terminal devices

101, 102, 103 and/or the server 104. Accordingly, the data processing apparatus provided by the embodiment of the present disclosure may also be disposed in a server or a server cluster different from the server 104 and capable of communicating with the

terminal devices

101, 102, 103 and/or the server 104.

It should be understood that the number of terminal devices, networks, and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for an implementation.

Fig. 2 schematically shows a flow chart of a data processing method according to an embodiment of the present disclosure.

As shown in fig. 2, the data processing method includes operations S210 to S230.

In operation S210, in response to detecting that information related to a first application in an application list is updated, an acquisition request is sent, where the application list is used to record information of at least one application running on a server, and the acquisition request is used to acquire a first acquisition tool corresponding to the first application.

According to the embodiment of the disclosure, the acquisition request of the acquisition tool of the relevant application program can be sent to the cloud platform according to the information update of the application program in the application program list.

According to an embodiment of the present disclosure, the first application may represent an application in which information in the application list is updated.

According to an embodiment of the present disclosure, the information of the application program may include, for example, id of the application program, version information, CPU configuration information, memory configuration information, storage configuration information, network bandwidth configuration information, and the like.

According to embodiments of the present disclosure, a collection tool, which may also be referred to as a collection client or collection agent, is a computing entity that can function continuously and autonomously in a distributed or collaborative system.

According to the embodiment of the disclosure, the data collection rule in the collection tool may be used to specify the data category of the application program, and the data collection rules of different collection tools are different, for example, the log, index, configuration information, and the like of the application program may be specified.

According to the embodiment of the disclosure, the acquisition tools can correspond to the application programs one by one, and the same acquisition tool can only acquire the data of the application program to which the same acquisition tool belongs.

According to the embodiment of the disclosure, when the version of the application program changes, the data category generated by the application program changes. The collection tool can execute different data collection rules according to the version information, so that the data collection of the application programs with different versions is completed.

In operation S220, in response to receiving the first collecting tool, first target data for the first application program is collected using the first collecting tool, wherein the first target data includes and at least one type information.

According to the embodiment of the disclosure, when the information of the first application program is updated, different data collection rules can be executed by the first collection tool according to the updated information, so as to complete the collection of the first target data related to the first application program.

According to the embodiment of the disclosure, the first target data can be sent to the corresponding database to be stored through the type information.

In operation S230, the first target data is transmitted to the target database according to the at least one type information, so that the target database stores the first target data by classification.

According to an embodiment of the present disclosure, the target databases may correspond to the type information one-to-one, and the first target data may include one or more types of information, i.e., the first target data may be stored to one or more target databases.

The method shown in fig. 2 is further described with reference to fig. 3-6 in conjunction with specific embodiments.

Fig. 3 schematically shows an application scenario diagram of a data processing method according to an embodiment of the present disclosure.

As shown in fig. 3, an application scenario diagram of the data processing method provided by the embodiment of the present disclosure may include: the management terminal 310, the local server 321, the local server 322.. local server 32n, the cloud platform 330, the topology association database 341, the log database 342, and the timing index database 343.

According to the embodiment of the present disclosure, various applications may be run on the

local servers

321, 322 …, 32n, and each local server may detect an update of related application information in the local application list, may then send an acquisition request to the cloud platform 330, and may receive an installation package or a download link from a related acquisition tool of the cloud platform 330; then each local server can respectively utilize related acquisition tools to acquire target data; and each local server can send the target data to the corresponding target database according to the type information of the target data, so that the topology association database 341, the log database 342, and the time sequence index database 343 are stored respectively.

According to the embodiment of the present disclosure, the

local servers

321, 322 … and 32n may further send respective heartbeat packets to the management terminal 310, so that the management terminal 310 sends the abnormal alarm information to a certain local server when not receiving the heartbeat packet of the local server again within a preset time period, so as to facilitate the local server to perform detection processing.

According to an embodiment of the present disclosure, the type information includes at least one of: log data, timing index data and topology association data; wherein the first target data further comprises tag information, the tag information comprising at least one of: identification information of the server, a collection time, and version information of the first collection tool.

According to an embodiment of the disclosure, the log data may record the procedural event data generated by updating the first application, and the log data may include the content of a user, time, device or application system, operation, and the like.

According to an embodiment of the present disclosure, the time-series index data may record a chronological data column of the same index, and the time-series index data may include data reflecting the degree of change by a numerical value.

According to an embodiment of the present disclosure, the topological correlation data may record interrelationships between the graphic elements such as points, lines, and planes, and the topological correlation data may include adjacency and connection relations between the graphic elements such as points, lines, and planes.

According to an embodiment of the present disclosure, the identification information of the server may represent a computer identifier of the operation server, and may be, for example, an IP address or the like.

According to an embodiment of the present disclosure, the collection time may represent a time at which collection of the first target data of the first application is completed using the first collection tool.

According to an embodiment of the present disclosure, the version information of the first collection tool may include an identifier of a specific application and a version of the application, for example, may be a port number or the like.

According to an embodiment of the present disclosure, the data processing method further includes:

transmitting the log original text data and the identification information to a log database under the condition that the first target data comprises log data; transmitting the time sequence index data and the identification information to a time sequence index database under the condition that the first target data comprises the time sequence index data; in the case where the first target data includes topology association data, the topology association data and the identification information are transmitted to a topology association database.

According to embodiments of the present disclosure, the log database may include a distributed NoSQL database that may be used to store log data.

According to embodiments of the present disclosure, the timing index database may include a distributed timing database, which may be used to store timing index data.

According to an embodiment of the disclosure, the topology association database may include a graph database, which may be used to store topology association data, such as network flow information collected by the probe.

According to the embodiment of the disclosure, data can be classified and stored according to the type information of the data by setting different databases, and the diversified requirements of the storage and the use modes of the collected data can be met for the classification processing of the collected data.

According to the embodiment of the disclosure, the heartbeat packet may represent a self-defined command word that regularly notifies the other party of its own status between the client and the server, and is transmitted at a certain time interval.

According to the embodiment of the present disclosure, the specific value of the preset time period may be flexibly set by a person skilled in the art according to the actual application situation, and the embodiment of the present disclosure does not limit the specific value of the preset time period.

According to the embodiment of the disclosure, for example, if the preset time period is 10s, the abnormal warning information may be generated when the heartbeat packet is not received again within 10s, and the abnormal warning information may be displayed to the relevant person in a manner of e.g. mail, short message, or the like.

According to the embodiment of the disclosure, the running state of each local server can be monitored in real time through the management terminal, so that the supervision efficiency is improved.

Fig. 4 schematically illustrates a flow chart of a method of unloading a second capture tool in accordance with an embodiment of the disclosure.

As shown in fig. 4, the method of unloading the second picking tool includes operations S410 to S440.

In operation S410, an application program running on a server is scanned.

According to the embodiment of the disclosure, the application program which is offline on the server can be monitored in real time by scanning the application program which is running on the server.

In operation S420, information of a second application that is not currently running is determined.

According to an embodiment of the present disclosure, the second application may include an application that is not currently running, for example, the second application may be an application that is automatically stopped according to a running timing, automatically stopped due to a running error, or actively stopped by an operation and maintenance person in a server, so as to determine id and version information of the offline application, and the like.

In operation S430, the second application is deleted in the application list.

According to the embodiment of the present disclosure, by deleting the second application program in the application program list, data collection of the second application program which is not running can be stopped.

In operation S440, a second collection tool corresponding to the second application program is uninstalled.

According to the embodiment of the disclosure, the second collection tool corresponding to the second application program which is not currently running is unloaded, so that the running space on the server can be released, and the performance of the server is improved.

Fig. 5 schematically illustrates a flow chart of a method of acquiring a first capture tool in accordance with an embodiment of the disclosure.

As shown in fig. 5, the method of acquiring the first collection tool includes operations S510 to S530.

In operation S510, an installation package or a download link of a first acquisition tool from a cloud platform is received, wherein the cloud platform includes a plurality of acquisition tools.

According to the embodiment of the disclosure, the cloud platform may include acquisition tools of various versions of an application program concerned by the operation and maintenance monitoring system, and the cloud platform may match and acquire a required acquisition tool for the server.

According to an embodiment of the present disclosure, an installation package may be a collection of files that can be decompressed by itself, and may include all files required for application installation.

According to the embodiment of the disclosure, when the installation package is operated, all files of the application program can be released to the hard disk, so that the subsequent work of modifying the registry, modifying the system setting, creating the shortcut and the like can be completed.

In operation S520, in response to receiving the download link, a random parameter is generated according to a storage capacity required when the download link is executed.

According to an embodiment of the present disclosure, the download link may be a computer program that performs information retrieval and downloads documents.

According to embodiments of the present disclosure, the storage capacity may represent the amount of binary information that the various files associated with the download link need to occupy in the server memory.

In operation S530, a first collection tool corresponding to the download link is obtained in a time-sharing manner according to the random parameter.

According to the embodiment of the disclosure, because the same application program is usually newly deployed on a batch of servers, or an application program is downloaded from a batch of servers at the same time, in order to avoid that the acquisition tools are downloaded from the same batch of local servers at the same time and the network transmission bandwidth is affected, the random parameter may be generated according to the size of the downloaded content, so that the same batch of servers implement the staggered downloading operation in a discrete time period according to the random parameter.

According to the embodiment of the disclosure, the transmission success rate and the downloading efficiency of the network bandwidth can be improved through the discretization of the synchronous script issuing.

Fig. 6 schematically shows an application scenario of a data processing method according to another embodiment of the present disclosure.

As shown in fig. 6, an application scenario diagram of a data processing method provided by another embodiment of the present disclosure may include: the system comprises a management terminal cluster 610, a local server 621, a local server 622 …, a local server 62n, a sensor 6211, a sensor 6221 …, a sensor 62n1, a discrete synchronizer 6212, a discrete synchronizer 62n2 …, a discrete synchronizer 62n2, a distributed agent library 630, a topological relation database cluster 641, a log database cluster 642 and a target database cluster 643.

According to the embodiment of the present disclosure, the

sensors

6211, 6221 …, the sensors 62n1 may be configured as a program group, each sensor embeds information collection for common data objects, and by including service process management tools such as system and hypervisor, applications running on the local server may be automatically scanned and sensed, so that the first application updated in the local server can be automatically discovered.

According to the embodiment of the present disclosure, when a new application program/new service is deployed on the local server 621, the local server 622 …, or the local server 62n is offline/stops service, the corresponding sensor may automatically sense the change of the service process, and synchronize the updated application information to the discrete synchronizer.

According to the embodiment of the disclosure, each sensor can also periodically send self-checking heartbeat information and sensing result information to the management terminal cluster.

According to the embodiment of the disclosure, the discrete synchronizers 6212 and 62n2 …, 62n2 may receive the latest application information sent by the sensor, compare the latest application information with the previous information, and when the information is updated, each discrete synchronizer may download a corresponding acquisition tool from the distributed agent library according to the updated application information and synchronize the information to the management terminal cluster.

According to the embodiment of the disclosure, each discrete synchronizer can also specify an output database and a corresponding label of the acquired data according to the current application information provided by each sensor and the instruction information received from the management terminal cluster, and the same data can support a plurality of output databases and a plurality of labels.

According to the embodiment of the disclosure, the distributed agent library 630 may include acquisition tools of various versions corresponding to an application program concerned by the operation and maintenance monitoring system, so that matching and acquisition of the required acquisition tools by various discrete synchronizers are facilitated, and the distributed agent library 630 can provide a load balancing function, so as to respond to access and download requests of batch servers.

According to the embodiment of the disclosure, the management terminal cluster 610 may receive the heartbeat and the sensing message sent by each sensor, and monitor the running state of each sensor and the active condition of the application; the updating messages sent by each discrete synchronizer can be received and compared with the information sent by the corresponding sensor, and the running state and the information synchronization accuracy of each discrete synchronizer can be monitored; the method can also send instructions to each discrete synchronizer to intervene or adjust and update the acquisition tool strategy and the acquisition data transmission strategy, can also acquire the information of the acquisition tool list in the distributed agent library 630, push a newly added acquisition tool to the distributed agent library 630, and send out overdue waste acquisition tools on the line of the distributed agent library 630; and abnormal alarm information can be provided for operation and maintenance monitoring personnel.

According to the embodiment of the present disclosure, the topology association database cluster 641, the log database cluster 642 and the index database cluster 643 are described above and are not described herein again.

Based on the data processing method, the disclosure also provides a data processing device. The apparatus will be described in detail below with reference to fig. 7.

Fig. 7 schematically shows a block diagram of a data processing apparatus according to an embodiment of the present disclosure.

As shown in fig. 7, the data processing apparatus 700 includes: a first transmission module 710, an acquisition module 720 and a second transmission module 730.

The first sending module 710 is configured to send an obtaining request in response to detecting that information of a first application in an application list is updated, where the application list is used to record information of at least one application running on a server, and the obtaining request is used to obtain a first collection tool corresponding to the first application. In an embodiment, the first sending module 710 may be configured to perform the operation S210 described above, which is not described herein again.

An acquisition module 720, configured to acquire, in response to receiving the first acquisition tool, first target data for the first application using the first acquisition tool, wherein the first target data includes at least one type information. In an embodiment, the acquisition module 720 may be configured to perform the operation S220 described above, which is not described herein again.

The second sending module 730 is configured to send the first target data to the target database according to the at least one type information, so that the target database stores the first target data in a classified manner. In an embodiment, the second sending module 730 may be configured to perform the operation S230 described above, which is not described herein again.

According to an embodiment of the present disclosure, the type information includes at least one of: log data, timing index data and topology association data;

wherein the first target data further comprises tag information, the tag information comprising at least one of: identification information of the server, a collection time, and version information of the first collection tool.

According to an embodiment of the present disclosure, the data processing apparatus 700 further includes: the device comprises a first transmission module, a second transmission module and a third transmission module.

The first transmission module is used for transmitting the log original text data and the identification information to the log database under the condition that the first target data comprises the log data.

And the second transmission module is used for transmitting the time sequence index data and the identification information to the time sequence index database under the condition that the first target data comprises the time sequence index data.

And the third transmission module is used for transmitting the topology association data and the identification information to the topology association database under the condition that the first target data comprises the topology association data.

According to an embodiment of the present disclosure, the data processing apparatus 700 further includes: and a third sending module.

And the third sending module is used for sending the heartbeat packet to the management terminal so that the management terminal generates abnormal alarm information under the condition that the management terminal does not receive the heartbeat packet again in the preset time period.

According to an embodiment of the present disclosure, the data processing apparatus 700 further includes: the device comprises a scanning module, a determining module, a deleting module and an unloading module.

And the scanning module is used for scanning the application program running on the server.

And the determining module is used for determining the information of the second application program which is not operated currently.

And the deleting module is used for deleting the second application program in the application program list.

And the unloading module is used for unloading a second acquisition tool corresponding to the second application program.

According to an embodiment of the present disclosure, the data processing apparatus 700 further includes: the device comprises a receiving module, a generating module and an obtaining module.

The receiving module is used for receiving an installation package or a download link of a first acquisition tool from a cloud platform, wherein the cloud platform comprises a plurality of acquisition tools.

And the generation module is used for responding to the received download link and generating random parameters according to the required storage capacity when the download link is executed.

And the acquisition module is used for acquiring a first acquisition tool corresponding to the download link in a time-sharing manner according to the random parameter.

Any number of modules, sub-modules, units, sub-units, or at least part of the functionality of any number thereof according to embodiments of the present disclosure may be implemented in one module. Any one or more of the modules, sub-modules, units, and sub-units according to the embodiments of the present disclosure may be implemented by being split into a plurality of modules. Any one or more of the modules, sub-modules, units, sub-units according to embodiments of the present disclosure may be implemented at least in part as a hardware circuit, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a system on a chip, a system on a substrate, a system on a package, an Application Specific Integrated Circuit (ASIC), or may be implemented in any other reasonable manner of hardware or firmware by integrating or packaging a circuit, or in any one of or a suitable combination of software, hardware, and firmware implementations. Alternatively, one or more of the modules, sub-modules, units, sub-units according to embodiments of the disclosure may be at least partially implemented as a computer program module, which when executed may perform the corresponding functions.

For example, any plurality of the first sending module 710, the collecting module 720 and the second sending module 730 may be combined and implemented in one module/unit/sub-unit, or any one of the modules/units/sub-units may be split into a plurality of modules/units/sub-units. Alternatively, at least part of the functionality of one or more of these modules/units/sub-units may be combined with at least part of the functionality of other modules/units/sub-units and implemented in one module/unit/sub-unit. According to an embodiment of the present disclosure, at least one of the first transmitting module 710, the collecting module 720 and the second transmitting module 730 may be implemented at least partially as a hardware circuit, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a system on a chip, a system on a substrate, a system on a package, an Application Specific Integrated Circuit (ASIC), or may be implemented by hardware or firmware in any other reasonable manner of integrating or packaging a circuit, or may be implemented by any one of three implementations of software, hardware and firmware, or any suitable combination of any of the three implementations. Alternatively, at least one of the first transmitting module 710, the acquiring module 720 and the second transmitting module 730 may be at least partially implemented as a computer program module, which when executed may perform a corresponding function.

It should be noted that, the data processing apparatus portion in the embodiment of the present disclosure corresponds to the data processing method portion in the embodiment of the present disclosure, and the description of the data processing apparatus portion specifically refers to the data processing method portion, which is not described herein again.

Fig. 8 schematically shows a block diagram of an electronic device adapted to implement a data processing method according to an embodiment of the present disclosure. The electronic device shown in fig. 8 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present disclosure.

As shown in fig. 8, a computer electronic device 800 according to an embodiment of the present disclosure includes a processor 801 which can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM)802 or a program loaded from a storage section 808 into a Random Access Memory (RAM) 803. The processor 801 may include, for example, a general purpose microprocessor (e.g., a CPU), an instruction set processor and/or associated chipset, and/or a special purpose microprocessor (e.g., an Application Specific Integrated Circuit (ASIC)), among others. The processor 801 may also include onboard memory for caching purposes. The processor 801 may include a single processing unit or multiple processing units for performing different actions of the method flows according to embodiments of the present disclosure.

In the RAM 803, various programs and data necessary for the operation of the electronic apparatus 800 are stored. The processor 801, the ROM 802, and the RAM 803 are connected to each other by a bus 804. The processor 801 performs various operations of the method flows according to the embodiments of the present disclosure by executing programs in the ROM 802 and/or RAM 803. Note that the programs may also be stored in one or more memories other than the ROM 802 and RAM 803. The processor 801 may also perform various operations of method flows according to embodiments of the present disclosure by executing programs stored in the one or more memories.

Electronic device 800 may also include input/output (I/O) interface 805, input/output (I/O) interface 805 also connected to bus 804, according to an embodiment of the present disclosure. Electronic device 800 may also include one or more of the following components connected to I/O interface 805: an input portion 806 including a keyboard, a mouse, and the like; an output section 807 including a signal such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage portion 808 including a hard disk and the like; and a communication section 809 including a network interface card such as a LAN card, a modem, or the like. The communication section 809 performs communication processing via a network such as the internet. A drive 810 is also connected to the I/O interface 805 as necessary. A removable medium 811 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 810 as necessary, so that a computer program read out therefrom is mounted on the storage section 808 as necessary.

According to embodiments of the present disclosure, method flows according to embodiments of the present disclosure may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable storage medium, the computer program containing program code for performing the method illustrated by the flow chart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication section 809 and/or installed from the removable medium 811. The computer program, when executed by the processor 801, performs the above-described functions defined in the system of the embodiments of the present disclosure. The systems, devices, apparatuses, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the present disclosure.

The present disclosure also provides a computer-readable storage medium, which may be embodied in the device/apparatus/system described in the above embodiments; or may exist separately and not be assembled into the device/apparatus/system. The computer-readable storage medium carries one or more programs which, when executed, implement the method according to an embodiment of the disclosure.

According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium. Examples may include, but are not limited to: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

For example, according to embodiments of the present disclosure, a computer-readable storage medium may include the ROM 802 and/or RAM 803 described above and/or one or more memories other than the ROM 802 and RAM 803.

Embodiments of the present disclosure also include a computer program product comprising a computer program containing program code for performing the method provided by the embodiments of the present disclosure, when the computer program product is run on an electronic device, the program code being adapted to cause the electronic device to carry out the data processing method provided by the embodiments of the present disclosure.

The computer program, when executed by the processor 801, performs the above-described functions defined in the system/apparatus of the embodiments of the present disclosure. The systems, apparatuses, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the present disclosure.

In one embodiment, the computer program may be hosted on a tangible storage medium such as an optical storage device, a magnetic storage device, or the like. In another embodiment, the computer program may also be transmitted in the form of a signal on a network medium, distributed, downloaded and installed via communication section 809, and/or installed from removable media 811. The computer program containing program code may be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the foregoing.

In accordance with embodiments of the present disclosure, program code for executing computer programs provided by embodiments of the present disclosure may be written in any combination of one or more programming languages, and in particular, these computer programs may be implemented using high level procedural and/or object oriented programming languages, and/or assembly/machine languages. The programming language includes, but is not limited to, programming languages such as Java, C + +, python, the "C" language, or the like. The program code may execute entirely on the user computing device, partly on the user device, partly on a remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., through the internet using an internet service provider).

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions. It will be appreciated by those skilled in the art that various combinations and/or combinations of the features recited in the various embodiments of the disclosure and/or the claims may be made even if such combinations or combinations are not explicitly recited in the disclosure. In particular, various combinations and/or combinations of the features recited in the various embodiments and/or claims of the present disclosure may be made without departing from the spirit or teaching of the present disclosure. All such combinations and/or associations are within the scope of the present disclosure.

The embodiments of the present disclosure have been described above. However, these examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although the embodiments are described separately above, this does not mean that the measures in the embodiments cannot be used in advantageous combination. The scope of the disclosure is defined by the appended claims and equivalents thereof. Various alternatives and modifications can be devised by those skilled in the art without departing from the scope of the present disclosure, and such alternatives and modifications are intended to be within the scope of the present disclosure.

Claims

1. A method of data processing, comprising:

in response to receiving the first collection tool, collecting, with the first collection tool, first target data for the first application, wherein the first target data includes and at least one type information; and

and sending the first target data to a target database according to the at least one type information so as to store the first target data in a classified manner by the target database.

2. The method of claim 1, wherein the type information comprises at least one of: log data, timing index data and topology association data;

3. The method of claim 2, wherein the sending the first target data to a target database according to the at least one type information, so that the target database stores the first target data by classification, comprises:

transmitting the log original text data and the identification information to a log database under the condition that the first target data comprises the log data;

transmitting the time series index data and the identification information to a time series index database if the first target data comprises the time series index data;

and transmitting the topology association data and the identification information to a topology association database under the condition that the first target data comprises the topology association data.

4. The method of any of claims 1-3, further comprising, prior to said sending an acquisition request in response to detecting that information associated with a first application in the list of applications is updated:

scanning an application program running on the server;

deleting the second application in the application list; and

and uninstalling a second acquisition tool corresponding to the second application program.

5. The method of any of claims 1-4, further comprising, prior to said collecting, with the first collection tool, first target data for the first application in response to receiving the first collection tool:

6. The method of claim 1, further comprising:

7. A data processing apparatus comprising:

the device comprises a first sending module, a first sending module and a second sending module, wherein the first sending module is used for sending an acquisition request in response to the fact that information of a first application program in an application program list is updated, the application program list is used for recording information of at least one application program running on a server, and the acquisition request is used for acquiring a first acquisition tool corresponding to the first application program;

a collection module for collecting first target data for the first application program using the first collection tool in response to receiving the first collection tool, wherein the first target data includes and at least one type information; and

and the second sending module is used for sending the first target data to a target database according to the at least one type information so as to be convenient for the target database to store the first target data in a classified manner.

8. An electronic device, comprising:

one or more processors;

a memory to store one or more instructions that,

wherein the one or more instructions, when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-6.

9. A computer readable storage medium having stored thereon executable instructions which, when executed by a processor, cause the processor to carry out the method of any one of claims 1 to 6.

10. A computer program product comprising computer executable instructions for implementing the method of any one of claims 1 to 6 when executed.