CN110673945A - Distributed task management method and management system - Google Patents
Distributed task management method and management system Download PDFInfo
- Publication number
- CN110673945A CN110673945A CN201810718590.XA CN201810718590A CN110673945A CN 110673945 A CN110673945 A CN 110673945A CN 201810718590 A CN201810718590 A CN 201810718590A CN 110673945 A CN110673945 A CN 110673945A
- Authority
- CN
- China
- Prior art keywords
- task
- node
- pieces
- distributed
- storing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5083—Techniques for rebalancing the load in a distributed system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
- G06F9/5038—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the execution order of a plurality of tasks, e.g. taking priority or time dependency constraints into consideration
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
An embodiment of the present invention provides a distributed task management system, including: a registry receiving a plurality of tasks and structuring and storing the plurality of tasks as a node data set; and the plurality of computing nodes read the node data in the node data set from the registry and execute corresponding tasks according to the node data. By structuring the task into a node data set and enabling the computing node to read node data from the data set, the computing node can read the node data as required, so that the performance pressure of the computing node is reduced, and the data processing efficiency is improved. The invention also provides a distributed task management system.
Description
Technical Field
The invention relates to the technical field of computers, in particular to a distributed task management method and a distributed task management system.
Background
Smart callout refers to the replenishment of goods from regional warehouses (RDCs) deployed in first-line cities to front warehouses (FDCs) located in second-line cities. This allows goods to be distributed across the country and suppliers may only need to deliver goods to regional warehouses (RDCs) rather than to warehouses throughout the country. The calculation of the replenishment quantity requires replenishment calculation for each SKU (Stock Keeping Unit), and since the data amount involved in the replenishment calculation based on SKUs is very large, the calculation task needs to be distributed to a plurality of servers and executed at the same time. In the prior art, an RPC (Remote Procedure Call) technology and a fragmentation method are two task allocation methods which are often adopted.
RPC technology uses remote method call methods such as JSF. As shown in fig. 1, on the master node master, a task is divided into a plurality of task pieces, and an RPC interface is called to place the plurality of task pieces on three computation servers, namely, Calc1-Calc3, for operation, so as to obtain a return result. The disadvantages of the RPC technique are: due to the limitation of network bandwidth and the like, the data volume carried by each RPC request must be proper, so for a task with large data volume, the task with large data volume is divided into a plurality of task requests, a huge number of task requests are generated instantly, and great performance pressure is caused on a computing server. For example, if a task request is distributed in groups of ten thousand pieces of data, 100 times of data needs to be distributed to 3 servers, each computing server has nearly 30 task requests instantaneously due to the very fast distribution process, and if the processing process of each task request is a very time-consuming work, the JVM (virtual machine) cannot be released, so that the computing server is overloaded and overheated, and the whole computing cluster is affected.
The fragmentation method, such as the elastic Job open source framework, fragments the data, and each data node obtains its task fragment for computation. The slicing method is relatively coarse-grained slicing, and is prone to data skew. For example, 300 pieces of data, corresponding to the above calculation servers Calc1-Calc3, each server will be divided into 100 ten thousand data at a time, and it is easy to make JVM burst.
Therefore, the inventors found that neither the RPC technique nor the fragmentation method described above has a problem of being unable to efficiently handle tasks of large data volumes.
Disclosure of Invention
In view of the above, embodiments of the present invention provide a distributed task management system and method, in which a computing node reads structured node data from a registry to improve data processing efficiency.
According to a first aspect of an embodiment of the present invention, a distributed task management method is provided, including:
receiving a plurality of task slices;
structuring and storing the plurality of task pieces as a node data set;
reading node data in the node data set; and
and executing corresponding tasks according to the node data.
Preferably, the method further comprises the following steps: dividing a task into a plurality of task pieces;
the structuring and storing the plurality of task tiles as a node data set comprises: each task slice constitutes a piece of node data.
Preferably, the plurality of task slices are structured as a tree structure, each node data being stored in a respective one of the tree nodes.
Preferably, the structuring and storing the plurality of task slices as a node data set further comprises: storing the IP addresses of the plurality of compute nodes in a tree node of the tree structure.
Preferably, the structuring and storing the plurality of task slices as a node data set further comprises: storing the execution state of each task slice in a tree node of the tree structure.
Preferably, the storing the execution state of each task slice in a tree node in the tree structure comprises:
and storing information of the task pieces to be executed, the task pieces failed to be executed, the task pieces successful to be executed and the task pieces being executed in tree nodes of the tree structure.
Preferably, when reading the task slice to be executed, adding the information of the task slice being executed in the corresponding tree node, when the task fails to be executed, adding the information of the task slice failing to be executed in the corresponding tree node, and after the task is executed, adding the information of the task slice successfully executed in the corresponding tree node.
Preferably, the method further comprises the following steps: and storing the task snapshot into a database in the task execution process.
Preferably, the method further comprises the following steps: and recovering the task according to the task snapshot.
According to a second aspect of the embodiments of the present invention, there is provided a distributed task management system, including:
a registry receiving a plurality of task pieces and structuring and storing the plurality of task pieces as a node data set;
and the plurality of computing nodes read the node data in the node data set from the registry and execute corresponding tasks according to the node data.
Preferably, the structuring and storing the plurality of task slices as a node data set comprises: each task slice constitutes a piece of node data.
Preferably, the plurality of task slices are structured as a tree structure, each node data being stored in a respective one of the tree nodes.
Preferably, the plurality of computing nodes are triggered to execute the corresponding tasks by a timer.
Preferably, the structuring and storing the plurality of task slices as a node data set further comprises: storing the IP addresses of the plurality of compute nodes in respective tree nodes of the tree structure.
Preferably, the structuring and storing the plurality of task slices as a node data set further comprises: storing the execution state of the task slices of the plurality of compute nodes in a tree node in the tree structure.
Preferably, storing the execution states of the task pieces of the plurality of compute nodes in the tree structure comprises:
and storing information of the task pieces to be executed, the task pieces failed to execute, the task pieces successful to execute and the task pieces being executed in the corresponding tree nodes.
Preferably, when the plurality of computing centers read the task pieces to be executed, the registry increases information of the task pieces being executed, when the plurality of computing centers fail to execute the tasks, the registry increases information of the task pieces that have failed to execute, and when the plurality of computing centers finish executing the tasks, the registry increases information of the task pieces that have succeeded in executing.
Preferably, the method further comprises the following steps: the main node receives a task and divides the task into a plurality of task pieces.
Preferably, the computing node stores the task snapshot in a database.
Preferably, the computing node restores the task according to the task snapshot.
Preferably, the registry is based on ZooKeeper.
According to a third aspect of embodiments of the present invention, there is provided a computer-readable storage medium storing computer instructions which, when executed, implement the distributed task management method described above.
According to a fourth aspect of the embodiments of the present invention, there is provided a distributed task management apparatus including:
a memory for storing computer instructions;
a processor coupled to the memory, the processor configured to perform the distributed task management method described above based on computer instructions stored by the memory.
The embodiment of the invention has the following advantages or beneficial effects: the task pieces are structured into node data sets which are stored in the registration center, and the computing nodes read the node data from the node data sets, so that the performance pressure of the computing nodes can be reduced, and the data processing efficiency is improved.
The preferred embodiments of the present invention have the following advantages or benefits: the node data is stored as a structure tree, and the execution state of the task piece is obtained by traversing the tree, so that the task execution condition can be monitored.
Another preferred embodiment of the present invention has the following advantages or benefits: the task snapshot is stored in the database, and the task execution is automatically recovered by using the task snapshot when the task fails, so that the reliability of the system can be improved.
Drawings
The above and other objects, features and advantages of the present invention will become more apparent by describing embodiments of the present invention with reference to the following drawings, in which:
FIG. 1 is a schematic diagram of the structure of a prior art RPC technique;
FIG. 2 is a diagram of a service cluster deployment in the prior art;
FIG. 3 is a flow diagram of a distributed task management method according to an embodiment of the invention;
FIG. 4 is a schematic structural diagram of a distributed task management system according to an embodiment of the present invention;
FIG. 5 is an exemplary diagram of a node data set of a distributed task management method according to an embodiment of the invention;
fig. 6 is a structural diagram of a distributed task management apparatus according to an embodiment of the present invention.
Detailed Description
The present invention will be described below based on examples, but the present invention is not limited to only these examples. In the following detailed description of the present invention, certain specific details are set forth. It will be apparent to one skilled in the art that the present invention may be practiced without these specific details. Well-known methods, procedures, and procedures have not been described in detail so as not to obscure the present invention. The figures are not necessarily drawn to scale.
As shown in fig. 2, a server cluster 10 including a plurality of servers has cluster components disposed on each server cluster. Consistency management services are provided for servers in a cluster through cluster components. Referring to FIG. 2, when a client sends a request to a cluster of servers, the cluster internally decides which server or servers ultimately handle the client's request, but the presentation to the client is the same view for all client clients. The cluster component, such as the ZooKeeper component, is a distributed, open-source distributed application program coordination service, and provides a consistency service for distributed applications, and the functions provided include: configuration maintenance, domain name service, distributed synchronization, group service, etc. ZooKeeper provides not only a performance efficient, functionally stable system but also a simple and easy-to-use interface to the user. With a cluster component such as ZooKeeper, the server cluster faces requests of multiple client clients in one whole.
Based on the server cluster, the method for distributed task management according to the embodiment of the invention is implemented.
Fig. 3 is a flowchart of a distributed task management method according to an embodiment of the present invention, which specifically includes the following steps.
In step S101, a plurality of task pieces are received.
In step S102, a plurality of task pieces are structured and stored as a node data set.
In step S103, node data in the node data set is read.
In step S104, a corresponding task is executed according to the node data.
Specifically, for the service cluster, a plurality of task slices are received, and the task slices may be from another node which divides the task into a plurality of task slices. Accordingly, the data that each task slice needs to process is reduced. For example, 300 ten thousand pieces of data are divided into 30 task pieces, and each task piece processes 10 ten thousand pieces of data. For example, if the replenishment quantity is calculated according to SKU as described in the background art, the related data is divided into a group of 10 ten thousand pieces to obtain a plurality of task pieces. These task slices are then stored as a structured node data set. Each task slice may store one node data correspondingly. The node data sets may be stored using data queues that are configured to have a first-in-first-out characteristic such that task pieces can only be picked up and processed serially. Thereby ensuring that the task pieces are not picked up repeatedly. And on one server in the node data set storage cluster, other servers sequentially obtain the node data from the node data set and execute corresponding task pieces according to the node data.
In the embodiment, the task pieces are structured into the node data sets to be stored, and the plurality of computing nodes respectively read the node data from the node data sets, so that the performance pressure of the computing nodes is reduced, and the data processing efficiency is improved. If the server cluster comprises the low-configuration high-configuration servers, each server can dynamically acquire node data according to the processing capacity and execute corresponding tasks, so that the low-configuration servers are protected, and the processing efficiency is considered.
The above-described embodiments are particularly advantageous for tasks that require processing large amounts of data. And after one task piece is executed, the next task piece is obtained, so that the memory resource of the server can be effectively protected. The plurality of servers acquire different numbers of task pieces according to the computing capacity of the servers, so that the low-configuration servers are effectively protected.
In an alternative embodiment, the user may specify the size of the task slice in advance, and accordingly, the task is divided, so that the problem of data skew is effectively solved.
FIG. 4 is a schematic diagram of a distributed task management system according to an embodiment of the present invention. The distributed task management system 20 includes a registry 300 and a plurality of computing nodes 200. The registry 300 receives a plurality of task pieces and structures and stores the plurality of task pieces as a node data set. The plurality of computing nodes 200 read node data in the node data set from the registry and perform corresponding tasks according to the node data.
Specifically, when the system works, the registry 300 receives a plurality of task pieces, structures the task pieces into node data sets, stores the node data sets in the registry, and the computing nodes 200 respectively read the node data from the registry 300 and execute corresponding tasks. In this process, the registry monitors the performance of tasks on the compute node 200 and records it in a structured node data set.
In the embodiment, the task pieces are structured into the node data sets and stored in the registration center, and the computing nodes read the node data from the node data sets, so that the performance pressure of the computing nodes is reduced, and the data processing efficiency is improved.
The distributed task management system 20 may further include a master node 100 and a Redis database 400. The master node 100 is used to divide a task into a plurality of task pieces. Redis database 400 is used to store process data. In one embodiment, the registry 300 receives a task, which is divided into task pieces by the master node 100. In another embodiment, the task is received by the master node 100, and after the task is divided into a plurality of task pieces, the task pieces are sent to the registry 300.
The registry, the computing nodes and the master nodes can be provided with deployment cluster components, and the cluster components are used for managing the computing nodes and the master nodes entering the cluster. At the same time, a cluster component is utilized to select a master node and a compute node to perform different functions. The read-write authority of the node data set in the registry 300 can be opened to a plurality of computing nodes 200, so that each computing node 200 can read the node data.
During the processing of a task slice by the compute node 200, a task snapshot may be saved to the Redis database 400, which is particularly effective for task slices with more steps. If a problem occurs in a certain step, the fast recovery from the snapshot data can be carried out in a certain step, and repeated calculation amount is avoided. And by the task snapshot, the system reliability is effectively improved.
Fig. 5 shows an example of the above node data set, and stores task slice information to represent a specific task in a tree structure, including task configuration nodes (task configurations), instance nodes (instances), and task slice nodes (task shards). The task configuration node stores metadata of a plurality of task pieces, the instance node stores IP addresses and host names of the computing node and the main node, and the task piece node stores execution states of the task pieces. The instances further include a master node and a calc node, the master node stores the IP address and the host name of the master node, the calc node stores the IP address and the host name of the computing node 200, the task shares further include nodes task queen, task processing, task failure and task completed, wherein the task queen stores the information of the task to be processed, the task processing stores the information of the task being processed, the task failure is the information of the task failing to be processed, and the task completed stores the information of the task successful to be processed. In the task slice node processing process, a task slice firstly waits for the computing node 200 to obtain from the task queen, once the task slice is obtained from the computing node 200, the task slice is transferred to the task processing, and records which computing node 200 (corresponding to the IP and the host) the task slice is currently processed by, if one task slice fails to process, the task slice is transferred to the task failover from the task processing, and records which computing node 200 the task slice fails to process at which step, and if the task slice succeeds in processing, the task slice is transferred to the task completed from the task processing. The Task slice in Task failover is further recorded to Task queen for reprocessing.
It can be seen that the above node data sets are stored in a tree structure. Of course, the embodiments of the present invention are not limited to the configuration information of such a storage structure, and may be stored in a relational database, for example.
The tree structure has the advantages that after the node data set is stored as the structure tree, the execution state of the task piece is obtained by traversing the tree, and the task execution condition is conveniently monitored.
Based on the above example of node data sets, when distributed computing fails, the following mechanism may be employed to handle the failed task slice. Specifically, when each compute node 200 reads a Task slice from Task Queen, a mapping is built under the Task Processing directory to indicate which server the Task slice is currently being executed by, e.g., Shard1- > 10.168.90.80. When the computing node 200 is disconnected from the registry, the master node 100 or the registry 300 is notified accordingly. At this time, the task piece corresponding to the disconnected connection is found, and the task piece is written into the taskfailover. Through the mechanism, even if the task fails, the task can be automatically recovered from the failure, so that the reliability of the system is ensured.
The master node 100 may also send heartbeat information to the computing node 200 at a timed rate to check the connection between the master node 100 and the computing node 200. Specifically, the master node 100 sends heartbeat information to the computing node 200, and if the computing node 200 has a heartbeat and is only disconnected from the registry, the master node 100 sends a command to stop computing to the computing node 200, and the computing node 200 stores a snapshot of task execution in Redis according to the command. The master node 100 or the registry 300 moves the corresponding task piece in the taskfailover to the task queen in the task shares. If a compute node is indeed hung, master node 100 or registry 300 takes the state of the compute task running from Redis, writes to task failover so that future machines can start directly from the failed step. Finally, the corresponding task slice in the task failover is moved to the task queen in the task names.
If the master node 100 fails, all servers connected with the registration center immediately elect the master node 100 by using the cluster characteristic, if the master node 100 fails when generating task shares, the elected master node 100 recovers according to the step of task slice processing of the last processing, if the elected master node 100 (previously used as a computing node 200) has a task executing, a snapshot of the task executing is stored in the Redis, and information in the task processing is moved to the task queue. When the task is finished, the master node 100 clears the intermediate information stored in the Redis and the directory information stored in the ZooKeeper, so that the next task is not required to be cleared again when being started. Through the mechanism, even if the task fails, the task can be automatically recovered from the failure, so that the reliability of the system is ensured.
Further, multithreading may also be used during the computation of the compute node 200 to provide per-data processing efficiency.
It should be noted that although the above embodiments illustrate some aspects of the present invention by way of ZooKeeper, the present invention is not limited thereto. Because other cluster management software or a new cluster management software developed autonomously may also be used to practice the distributed task management method provided by the embodiment of the present invention.
Fig. 6 is a structural diagram of a distributed task management apparatus according to an embodiment of the present invention. The apparatus shown in fig. 6 is only an example and should not limit the functionality and scope of use of embodiments of the present invention in any way.
Referring to fig. 6, the distributed task management apparatus includes a processor 601, a memory 602, and an input-output device 603 connected by a bus. The memory 602 includes a Read Only Memory (ROM) and a Random Access Memory (RAM), and various computer instructions and data required to perform system functions are stored in the memory 602 and read by the processor 601 from the memory 602 to perform various appropriate actions and processes. An input/output device including an input portion of a keyboard, a mouse, and the like; an output section including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage section including a hard disk and the like; and a communication section including a network interface card such as a LAN card, a modem, or the like. The memory 602 also stores the following computer instructions to perform the operations specified by the distributed task management method of embodiments of the present invention: receiving a plurality of task slices; structuring and storing the plurality of task pieces as a node data set; reading node data in the node data set; and executing corresponding tasks according to the node data.
Accordingly, embodiments of the present invention provide a computer-readable storage medium storing computer instructions, which, when executed, implement the operations specified by the distributed task management method described above.
The flowcharts and block diagrams in the figures and block diagrams illustrate the possible architectures, functions, and operations of the systems, methods, and apparatuses according to the embodiments of the present invention, and may represent a module, a program segment, or merely a code segment, which is an executable instruction for implementing a specified logical function. It should also be noted that the executable instructions that implement the specified logical functions may be recombined to create new modules and program segments. The blocks of the drawings, and the order of the blocks, are thus provided to better illustrate the processes and steps of the embodiments and should not be taken as limiting the invention itself.
The various modules or units of the system may be implemented in hardware, firmware or software. The software includes, for example, a code program formed using various programming languages such as JAVA, C/C + +/C #, SQL, and the like. Although the steps and sequence of steps of the embodiments of the present invention are presented in method and method diagrams, the executable instructions of the steps implementing the specified logical functions may be re-combined to create new steps. The sequence of the steps should not be limited to the sequence of the steps in the method and the method illustrations, and can be modified at any time according to the functional requirements. Such as performing some of the steps in parallel or in reverse order.
Systems and methods according to the present invention may be deployed on a single server or on multiple servers. For example, different modules may be deployed on different servers, respectively, to form a dedicated server. Alternatively, the same functional unit, module or system may be deployed in a distributed fashion across multiple servers to relieve load stress. The server includes but is not limited to a plurality of PCs, PC servers, blades, supercomputers, etc. on the same local area network and connected via the Internet.
The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (23)
1. A distributed task management method, comprising:
receiving a plurality of task slices;
structuring and storing the plurality of task pieces as a node data set;
reading node data in the node data set; and
and executing corresponding tasks according to the node data.
2. The distributed task management method of claim 1, further comprising: dividing a task into a plurality of task pieces;
the structuring and storing the plurality of task tiles as a node data set comprises: each task slice constitutes a piece of node data.
3. The distributed task management method of claim 2, wherein the plurality of task slices are structured as a tree structure, and wherein each node data is stored in a corresponding one of the tree nodes.
4. The distributed task management method of claim 3, wherein the structuring and storing the plurality of task tiles as a node data set further comprises: storing the IP addresses of the plurality of compute nodes in a tree node of the tree structure.
5. The distributed task management method of claim 3, wherein the structuring and storing the plurality of task tiles as a node data set further comprises: storing the execution state of each task slice in a tree node of the tree structure.
6. The distributed task management method of claim 5, wherein storing the execution state of each task slice in a tree node in the tree structure comprises:
and storing information of the task pieces to be executed, the task pieces failed to be executed, the task pieces successful to be executed and the task pieces being executed in tree nodes of the tree structure.
7. The distributed task management method according to claim 6, wherein when reading a task slice to be executed, information of the task slice being executed is added to the corresponding tree node, when the task fails to be executed, information of the task slice failing to be executed is added to the corresponding tree node, and when the task is finished, information of the task slice successfully executed is added to the corresponding tree node.
8. The distributed task management method of claim 1, further comprising: and storing the task snapshot into a database in the task execution process.
9. The distributed task management method of claim 1, further comprising: and recovering the task according to the task snapshot.
10. A distributed task management system, comprising:
a registry receiving a plurality of task pieces and structuring and storing the plurality of task pieces as a node data set;
and the plurality of computing nodes read the node data in the node data set from the registry and execute corresponding tasks according to the node data.
11. The distributed task management system of claim 10, wherein the structuring and storing the plurality of task tiles as a node data set comprises: each task slice constitutes a piece of node data.
12. The distributed task management system of claim 11, wherein the plurality of task tiles are structured as a tree structure, each node data being stored in a respective one of the tree nodes.
13. The distributed task management method of claim 10, wherein the plurality of computing nodes are triggered to execute the corresponding tasks by a timer.
14. The distributed task management system of claim 10, wherein the structuring and storing the plurality of task tiles as a node data set further comprises: storing the IP addresses of the plurality of compute nodes in respective tree nodes of the tree structure.
15. The distributed task management system of claim 12, wherein the structuring and storing the plurality of task tiles as a node data set further comprises: storing the execution state of the task slices of the plurality of compute nodes in a tree node in the tree structure.
16. The distributed task management system of claim 15, wherein storing the execution state of the task slices of the plurality of compute nodes in a tree node in the tree structure comprises:
and storing information of the task pieces to be executed, the task pieces failed to execute, the task pieces successful to execute and the task pieces being executed in the corresponding tree nodes.
17. The distributed task management system of claim 16, wherein when the plurality of computing centers read the task pieces to be executed, the registry increases information of the task pieces being executed, when the plurality of computing centers fail to execute the task, the registry increases information of the task pieces that have failed to execute, and when the plurality of computing centers finish executing the task, the registry increases information of the task pieces that have succeeded in executing.
18. The distributed task management system of claim 10, further comprising: the main node receives a task and divides the task into a plurality of task pieces.
19. The distributed task management system of claim 10, wherein the compute node stores a task snapshot in a database.
20. The distributed task management system of claim 10, wherein the compute node resumes the task from a task snapshot.
21. The distributed task management system of claim 10, wherein the registry is based on ZooKeeper.
22. A computer-readable storage medium storing computer instructions which, when executed, implement a distributed task management method according to any one of claims 1 to 9.
23. A distributed task management apparatus, comprising:
a memory for storing computer instructions;
a processor coupled to the memory, the processor configured to perform implementing the distributed task management method of any of claims 1-9 based on computer instructions stored by the memory.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810718590.XA CN110673945A (en) | 2018-07-03 | 2018-07-03 | Distributed task management method and management system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810718590.XA CN110673945A (en) | 2018-07-03 | 2018-07-03 | Distributed task management method and management system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110673945A true CN110673945A (en) | 2020-01-10 |
Family
ID=69065396
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810718590.XA Pending CN110673945A (en) | 2018-07-03 | 2018-07-03 | Distributed task management method and management system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110673945A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112685047A (en) * | 2020-12-31 | 2021-04-20 | 杭州当虹科技股份有限公司 | Rapid analysis system based on large file |
| CN113177824A (en) * | 2021-05-06 | 2021-07-27 | 北京沃东天骏信息技术有限公司 | Replenishment task processing method, replenishment task processing device, computer system and readable storage medium |
| CN114327817A (en) * | 2021-12-22 | 2022-04-12 | 马上消费金融股份有限公司 | Task fragmentation method and device and electronic equipment |
| CN117076555A (en) * | 2023-05-08 | 2023-11-17 | 芜湖本初子午信息技术有限公司 | Distributed task management system and method based on calculation |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101986272A (en) * | 2010-11-05 | 2011-03-16 | 北京大学 | Task scheduling method under cloud computing environment |
| CN102073546A (en) * | 2010-12-13 | 2011-05-25 | 北京航空航天大学 | Task-dynamic dispatching method under distributed computation mode in cloud computing environment |
| CN106980678A (en) * | 2017-03-30 | 2017-07-25 | 温馨港网络信息科技（苏州）有限公司 | Data analysing method and system based on zookeeper technologies |
| CN107046510A (en) * | 2017-01-13 | 2017-08-15 | 广西电网有限责任公司电力科学研究院 | A kind of node and its system of composition suitable for distributed computing system |
| CN110414961A (en) * | 2019-06-21 | 2019-11-05 | 深圳壹账通智能科技有限公司 | Prevent transfer account method, device, equipment and the storage medium of the track transactions side of producing |
| CN112699964A (en) * | 2021-01-13 | 2021-04-23 | 成都链安科技有限公司 | Model construction method, system, device, medium and transaction identity identification method |
-
2018
- 2018-07-03 CN CN201810718590.XA patent/CN110673945A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101986272A (en) * | 2010-11-05 | 2011-03-16 | 北京大学 | Task scheduling method under cloud computing environment |
| CN102073546A (en) * | 2010-12-13 | 2011-05-25 | 北京航空航天大学 | Task-dynamic dispatching method under distributed computation mode in cloud computing environment |
| CN107046510A (en) * | 2017-01-13 | 2017-08-15 | 广西电网有限责任公司电力科学研究院 | A kind of node and its system of composition suitable for distributed computing system |
| CN106980678A (en) * | 2017-03-30 | 2017-07-25 | 温馨港网络信息科技（苏州）有限公司 | Data analysing method and system based on zookeeper technologies |
| CN110414961A (en) * | 2019-06-21 | 2019-11-05 | 深圳壹账通智能科技有限公司 | Prevent transfer account method, device, equipment and the storage medium of the track transactions side of producing |
| CN112699964A (en) * | 2021-01-13 | 2021-04-23 | 成都链安科技有限公司 | Model construction method, system, device, medium and transaction identity identification method |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112685047A (en) * | 2020-12-31 | 2021-04-20 | 杭州当虹科技股份有限公司 | Rapid analysis system based on large file |
| CN113177824A (en) * | 2021-05-06 | 2021-07-27 | 北京沃东天骏信息技术有限公司 | Replenishment task processing method, replenishment task processing device, computer system and readable storage medium |
| CN114327817A (en) * | 2021-12-22 | 2022-04-12 | 马上消费金融股份有限公司 | Task fragmentation method and device and electronic equipment |
| CN117076555A (en) * | 2023-05-08 | 2023-11-17 | 芜湖本初子午信息技术有限公司 | Distributed task management system and method based on calculation |
| CN117076555B (en) * | 2023-05-08 | 2024-03-22 | 深圳市优友网络科技有限公司 | Distributed task management system and method based on calculation |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11797395B2 (en) | Application migration between environments | |
| US11074143B2 (en) | Data backup and disaster recovery between environments | |
| CN111480154B (en) | Method, system, and medium for batch data ingestion | |
| CN106453564B (en) | The method, apparatus and system of elastic cloud distribution magnanimity request processing | |
| US10942812B2 (en) | System and method for building a point-in-time snapshot of an eventually-consistent data store | |
| US9063790B2 (en) | System and method for performing distributed parallel processing tasks in a spot market | |
| US11663085B2 (en) | Application backup and management | |
| CN111488241B (en) | Method and system for realizing agent-free backup and recovery operation in container arrangement platform | |
| US8656387B2 (en) | Method and system for workload distributing and processing across a network of replicated virtual machines | |
| US8112659B2 (en) | Reducing recovery time for business organizations in case of disasters | |
| CN110673945A (en) | Distributed task management method and management system | |
| US9501544B1 (en) | Federated backup of cluster shared volumes | |
| US11119829B2 (en) | On-demand provisioning of customized developer environments | |
| US10860427B1 (en) | Data protection in a large-scale cluster environment | |
| US10616313B2 (en) | Scalable monitoring of long running multi-step data intensive workloads | |
| CN105871603A (en) | Failure recovery system and method of real-time streaming data processing based on memory data grid | |
| US10901860B2 (en) | Automated development of recovery plans | |
| JP2018535456A (en) | System and method for starting application servers in parallel | |
| Chen et al. | Adaptive data replicas management based on active data-centric framework in cloud environment | |
| Putri et al. | Implementation of resilience as a service for parallel computing | |
| Cermak et al. | Achieving high availability in D-Bobox | |
| CN113806093A (en) | Distribution calculation method of content tag calculation task and related device | |
| CN118939472A (en) | Data management method and related equipment | |
| CN113031852A (en) | Data processing method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |