CN107295106B

CN107295106B - Message data service cluster

Info

Publication number: CN107295106B
Application number: CN201710637641.1A
Authority: CN
Inventors: 胡悦; 吴文龙
Original assignee: Hangzhou Duomai Electronic Commerce Co ltd
Current assignee: Hangzhou Duomai Electronic Commerce Co ltd
Priority date: 2017-07-31
Filing date: 2017-07-31
Publication date: 2020-08-14
Anticipated expiration: 2037-07-31
Also published as: CN107295106A

Abstract

The invention relates to the field of message processing, in particular to a message data service cluster which comprises a theme zone for storing theme message data, wherein the theme zone comprises different service ends distributed in the service cluster; the method is characterized in that: the theme message data is stored in the service cluster in a memory mode or a file mode; in the memory mode, the theme zone receives the theme message data generated by the production end and maps the theme message data into the memory of the service end; and in the file mode, the theme partition receives the theme message data generated by the production end and stores the theme message data in a hard disk of the server end.

Description

Message data service cluster

Technical Field

The invention relates to the field of message processing, in particular to a message data service cluster.

Background

There are many message systems currently used for message (e.g., log) processing, and distributed message systems are more popular.

A distributed messaging system framework is shown in fig. 1. The system comprises a Producer (PD), an Agent (CS), a Consumer (CS) and a third-party management cluster, wherein each role can be multiple. The Producer sends the message to the Agent, the message is stored in the Agent in a persistent mode, and the Consumer obtains the message from the Agent for processing. The third party management cluster is used for storing some state information of the Producer, the Consumer and the Agent.

The distributed messaging system manages messages based on the Topic of the message (Topic). The storage in the message storage device is also based on the subject. And is persisted in a file mode in the hard disk of the server. This type of theme message data is not convenient to store using a file. Firstly, file storage is designed for messages with relatively long life cycles, and outdated data cannot be eliminated in time; secondly, for file storage, the data that can be consumed must be the subject message data that has landed (i.e. persisted to the hard disk with a valid logical version number), and there is a time difference between the subject message data being received and landed in the file storage mode, and the subject message data cannot be consumed in time.

Disclosure of Invention

In order to solve the technical problem, the invention provides a message data service cluster, which comprises a theme zone for storing theme message data, wherein the theme zone comprises different service ends distributed in the service cluster; the method is characterized in that: the theme message data is stored in the service cluster in a memory mode or a file mode; in the memory mode, the theme zone receives the theme message data generated by the production end and maps the theme message data into the memory of the service end; and in the file mode, the theme partition receives the theme message data generated by the production end and stores the theme message data in a hard disk of the server end.

Preferably, the topic partition comprises a data paragraph, and the data paragraph comprises the topic message data and an index file; the index file records a one-to-one mapping relation between the logical version number of the subject message data and the physical offset of the main message data.

Preferably, the logic version number reflects the sequence of the theme message data arriving at the service cluster.

Preferably, the subject partition includes a primary copy and a secondary copy distributed in different service terminals of the service cluster, the secondary copy is a redundant backup of the primary copy, the subject partition receives subject message data from the production terminal through the primary copy, and the subject partition responds to a consumption request of the consumption terminal through the primary copy.

Preferably, in the memory mode, the subject partition includes a data paragraph skip table, and the data paragraph skip table includes a plurality of data layers; the number of nodes of the next data layer is greater than that of nodes of the previous data layer, the next data layer comprises all nodes of the previous data layer, and the data layer positioned at the bottom layer comprises all nodes of the data paragraph skip table; the node comprises paragraph data comprising the subject message data; the node includes its pointer data in the next data layer, pointer data of the next node of the data layer where the node is located.

Preferably, in the file mode, the body partition records a range of physical offsets between the data passage and the subject message data in the data passage.

Preferably, the main copy records a valid logic version number of the latest received subject message data and a cache logic version number of the main message data cached in the memory.

Preferably, the production end polls the main copy of each theme zone to store the generated theme message data in each theme zone in a distributed manner.

Preferably, the consuming side polls each topic partition to consume the topic message data.

Preferably, the topic partition allows at most one consuming side in the same consuming cluster to consume the topic message data thereof at the same time.

Drawings

FIG. 1 is a prior art messaging system framework.

Fig. 2 prior art message system consumption pattern.

Fig. 3 is a distributed messaging system framework of the present invention.

FIG. 4 is a distributed message system consumption pattern of the present invention.

Fig. 5 is a diagram of a data paragraph skip table structure according to the present invention.

Detailed Description

The following specific examples are given by way of illustration only and not by way of limitation, and it will be apparent to those skilled in the art from this disclosure that various changes and modifications can be made in the examples without inventive faculty, and yet still be protected by the scope of the claims.

Example one

Fig. 3 shows a distributed message system framework of the present invention, which includes a production cluster for producing subject message data, a service cluster for storing the subject message data, a consumption cluster for consuming the subject message data, and a management cluster for managing the consumption cluster and the service cluster. The service cluster comprises a subject partition for storing subject message data, the subject partition comprises a master copy (Leader) and a slave copy (Follower) distributed in different service ends of the service cluster, and the slave copy is a redundant backup of the master copy. The service cluster is composed of one or more Agent servers (service terminals), and each service terminal is responsible for external service or data redundancy of a plurality of theme partitions. The server provides the submitting/consuming/partitioning addressing function of the message to the outside. One of the service terminals (agents) in the same subject partition is the master copy, and the rest are the slave copies. The Leader of each topic partition is distributed among the agents of the service cluster, and the same is true of the Follower. A producing end (Producer) in the production cluster accesses the main copy of the subject partition to store the subject message data generated by the producing end, and a consuming end (Consumer) in the consuming cluster accesses the main copy of the subject partition to consume the message data of the subject partition. The production end pushes the message to the server end to enable the message to be sent to the server end to be stored at the highest speed; the consumer can consume the message at a proper speed according to the consumption capacity of the consumer by the aid of the pull message of the server, and the possibility of network congestion is reduced.

The management cluster detects whether the server side and the consumption side are alive or not by detecting heartbeat signals of the server side and the consumption side, and addresses the server side according to the theme and the theme zone. Subject partition information under each subject is maintained, including subject partition addressing and master and slave copy information for the subject partition. And maintaining information of each server, including the number of master copies and the number of slave copies of the owned subject partition. When the information of the master copy and the slave copy of a certain theme partition is changed, the corresponding master copy or slave copy is informed.

Logical structure of main body message file

The theme message data is persisted in the hard disk of the server in the form of a theme message file. A topic may logically be considered a queue, and each consumption must specify its topic. I.e. it has to be indicated in which queue this message is put. In order to enable the throughput rate of the message system to be expanded horizontally, the theme message is physically divided into one or more theme partitions, each theme partition physically corresponds to a theme message folder, and all messages and index files of the theme partition are stored in the theme message folder. Specifically, the structure of a theme message file is as follows:

1. the message data are distinguished according to the Topic (Topic), and the messages in the same class belong to the same Topic (Topic). Topic message data refers to message data belonging to the same topic. The subject of the message data needs to be divided as required when in use.

2. The theme is divided into a plurality of theme partitions, the number of the partitions needs to be specified when the theme is created, and the theme partitions can be expanded after the theme is created.

3. Each topic partition is divided into a plurality of data segments (segments) according to the actual situation (namely the data size of the message data of the topic). Each data segment is approximately equal in size and each data segment is in one-to-one correspondence with an index file. The size of the data paragraph needs to be determined when the topic is created, and can also be changed after the topic is created.

4. Subject message data within each subject partition each subject message data is appended to the subject partition and assigned a logical version number in the order in which it arrives at the server, the logical version number being successively incremented. Each logical version number uniquely identifies the subject message data to which it corresponds and corresponds to the physical offset of the subject message data at the server. The subject message data is written into the disk sequentially, and the efficiency is very high.

Creation of a theme

Theme zone data and target service instances are specified when creating a theme. Theme creation is performed by sending commands to these service instances. The specific process is as follows:

1. the theme creation tool obtains Agent information for the service cluster from a third party management cluster (e.g., zookeeper). The Agent information mainly comprises the number of the Leader and the Follower of the theme partition owned by each Agent.

2. Take the example of creating a theme with 4 partitions and 3 copies. The theme creation tool selects the 3 least loaded servers (agents) as the 3 copies of the first partition of the new theme based on the load. The several agents are then notified to create the theme zone.

3. After the three agents successfully create the partition copy, entering a process of electing a master copy (Leader) (the process of electing the master copy refers to the disaster recovery strategy part).

4. The theme creation tool detects the creation of the first theme zone of the theme through the management cluster. If the creation is successful and the primary replica is enumerated, then the creation of the first subject partition is successful. Steps 2-4 are then repeated to create the remaining three subject partitions.

If the selected Agent is offline in the process of creating the Topic partition, the following conditions are divided into: if none of the selected agents are alive (i.e., all selected agents are down), then steps 2-4 are re-executed to recreate the new partition. If the selected Agent is still in stock, waiting for the successful creation of the Topic by checking the status of the management cluster.

Message redundancy backup

The main copy (Leader) of the subject partition is used as a receiving and consuming center of the message and directly provides external services. Several slave copies (Follower) acquire messages from the master copy (Leader) and update local message combinations. The master receives and stores the subject message data from the Producer (Producer) and updates the slave so that the slave is synchronized with the master. And the master/slave copy reports the minimum logic version number (minimum effective logic version number) of the master/slave copy and the received maximum logic version number (maximum effective logic version number) to the management cluster at regular time.

The master replica of each subject partition needs to maintain a set of slave replicas (a set of keep-alive replicas). The slave replicas within the set have updated message data relative to other slave replicas. When the number of the slave copies in the synchronous copy set is kept smaller than the minimum number of the synchronous copies, the master copy does not provide the service for receiving the message to the outside any more (note that only the topic partition under the topic does not provide the service for receiving the message any more, but the service end in the whole service cluster does not provide the service for receiving the message).

The service end (Agent) provides three guarantee levels for the production end (Producer) to send messages:

A. the master copy immediately sends information of successful message submission to the production end after receiving the theme message data sent by the production end in the level of UNSAFE;

B. in a FAST (FAST) level, a master copy synchronizes to all slave copies after receiving subject message data sent by a production end, and sends message submission success information to the production end after receiving determined synchronization information of a first slave copy;

C. and in the Safety (SAFE) level, the master copy synchronizes to all the slave copies after receiving the subject message data sent by the production end, and sends message submission success information to the production end after receiving the determined synchronization information of the slave copies with the number greater than or equal to the minimum number of the synchronization copies. And all slave copies that reply with the determined synchronization information are elements in the set of kept synchronized copies.

At the security level, a piece of subject message data is considered committed only if all of the secondary copies in the set of synchronized copies have been copied. The data loss caused by offline of partial data written into the master copy but not written into the slave copy is avoided, so that the data cannot be consumed by a consumption end. The data security and the message system throughput rate are well balanced. The secondary copy can copy data from the primary copy in batches, so that the copying performance is greatly improved, and the difference of data synchronization in the secondary copy and the primary copy is greatly reduced.

Disaster recovery strategy

In order to prevent the topic partition from being unavailable when the primary copy is offline, the disaster recovery strategy is adopted:

1. the service end in the service cluster needs to maintain a routing table, which indicates which service ends are available in a certain theme zone of a certain current theme. The routing table may also be used for service addressing, the updating of which is taken care of by the management cluster.

2. When the master copy goes offline, the new master copy election process is as follows: keeping the slave copies in the synchronous copy set to report the maximum logic version number (the maximum effective logic version number) of the slave copies to the management cluster; and selecting the slave copy corresponding to the maximum logic version number as a master copy according to the respective maximum logic version numbers (maximum valid logic version numbers) reported by all the slave copies in the copy synchronization set, and reporting the master copy to the management cluster. And if the number of the slave copies with the maximum logical version number is multiple, selecting the slave copy which is firstly reported to the management cluster as the master copy.

3. If a slave replica goes offline and the slave replica no longer maintains the set of synchronized replicas, then the offline of the slave replica has no effect on the entire service cluster.

4. If a slave replica in the set of kept synchronized replicas goes offline, the master replica needs to delete the slave replica from the set of kept synchronized replicas and report to the third party management cluster. The subject partition stops servicing when the number of slave replicas in the set of kept synchronized replicas is less than the minimum number of synchronized replicas.

Processing method of message system

After determining the subject, the production end tries to connect the master copies of all the subject districts of the subject, and then sequentially submits the messages in a polling mode.

Like a log, binlog of mysql, and the like. In this embodiment, the server topic message data is stored by using a hard disk file as a storage carrier. The theme message data has the characteristics of large volume and long storage time.

Hard disk IO is generally slow, and if the writing is performed little by little, such as several bytes at a time or random writing, the delay caused by the hard disk is considerable, and finally, the response speed and throughput of the application are reduced.

Therefore, in this embodiment, a mode that the memory buffers a batch of data first and then writes the hard disks in batch order is adopted to improve throughput.

With memory caching messages, there are typically two parameters to specify. The first is the size of the cache, and the second is the maximum time interval to flush the disk. It is necessary for the server to specify these two parameters when creating the theme. These two parameters may also be modified after the creation of the theme. Two parameters representing the message volume are included for the server: the effective logic version number of the message is the latest logic version number of the subject message data which is durably stored to the hard disk, and is visible to the consumption end; and the caching logic version number of the message is the logic version number of the latest theme message data received by the main copy of the theme partition and is invisible to the consumption end.

FIG. 4 shows a topic consumption model named A1, the topic having two topic partitions, P1 and P2. The message processing in P1 and P2 belong to the same subject and have no relation, and are completely different. Each subject partition is provided with a master copy and a slave copy for redundant data, and data safety is guaranteed. A routing table is required to be maintained inside each topic partition, and the valid logical version number of the messages in the master copy and the slave copy of the topic partition and the consumption group information of the master copy are maintained in the routing table. Each topic partition allows at most one consuming side in the same consuming cluster to consume the topic message data at the same time.

The valid logical version numbers of the messages of the master and slave copies of each subject partition are reported to the management cluster periodically (e.g., within 10 seconds) for later monitoring. When consuming, the consuming end in a certain consuming cluster only consumes the subject message data from the primary copy of the subject partition, and the primary copy records the consumption level of the consuming cluster (i.e. the logic version number of the consumed subject message data) to ensure that the consuming end in the same consuming cluster does not repeatedly consume the subject message data. Meanwhile, as the subject message data is consumed only from the primary copy of the subject partition, the consumption level does not need to be synchronized to other servers under normal conditions, and only the primary copy needs to maintain the consumption level locally. And the consumption end polls each theme partition to achieve the effect of consumption data with balanced load. The consumption level maintained by the master copy is reported to the management cluster periodically so as to avoid repeated consumption as far as possible after monitoring and reselecting the master copy of the theme zone. The consumption mode of the consumer can be two types:

A. in FAST consumption mode (FAST), after the primary copy sends the subject message data to the consumption end, the consumption level information of the consumption cluster where the consumption end is located is immediately updated.

B. And in a safety mode (SAFE), after the main copy sends the theme message data to the consumption end and receives the receiving confirmation information of the consumption end, the consumption level of the consumption cluster where the consumption end is located is updated. In this mode, a response timeout time needs to be established, and when the response timeout time is exceeded, the consumption level of the consumption cluster is updated by the primary copy even if the acknowledgement information of the consumption end is not received.

When the theme partition of the theme to which the server belongs changes, the server notifies the production end connected with the server of the change situation of the theme partition (such as the expansion of the theme partition). The production end dynamically adds and validates the theme partition and checks whether the off-line theme partition is on-line again or not. And the production end can sense the expansion of the theme partition and the offline of the main copy of the main partition. When the main copy of one theme partition is offline, the production end temporarily submits the theme message data to the main copies of other theme partitions which are not offline. And after the main copy of the offline subject partition is on-line again or a new main copy is elected, submitting a message to the subject partition. When one subject partition is in an unavailable state (i.e., the number of slave copies of the set of kept synchronized copies is less than the minimum number of synchronized copies), the production end submits the message to the subject partition in the remaining available state.

When a consuming side joins or goes offline, the consuming cluster needs to redistribute the partitions to achieve the purpose that the theme is normally consumed. The method specifically comprises the following steps: the server records the consumption level (i.e. the logic version number of the consumed subject message data) and the load level (i.e. the number of consumed subject partitions) of the consumption end in a certain consumption cluster of the subject. When the consumption end is on line, the primary copy of a certain theme partition of the theme is randomly selected for connection. The main copy of the connected subject partition is judged according to the load level of the current consumption cluster:

and if the theme partition managed by the primary copy is not consumed by any other consuming side which belongs to the same consuming cluster with the consuming side, directly accepting the connection request of the consuming side. And judging whether the consuming end needs to be connected with other theme partitions or not according to the consuming cluster load, and if so, informing the consuming end that redundant partitions need to be connected.

If the primary copy is already consumed by a certain consumer of the same consumption cluster, the primary copy needs to perform corresponding processing according to the consumption level of the consumer: if the consuming end only consumes the theme partition, the main copy returns a reasonable theme partition (the reasonable theme partition refers to whether the number of the consumed main partition of the consuming cluster in the theme partition is within an allowable range) to the consuming end and informs the consuming end of the connection information (such as the connection address) of the main copy to the current consumption; if the consumption end consumes a plurality of theme partitions, the master copy informs the consumption end of disconnecting and accepting the connection request of the consumption end.

When a consuming end goes offline, a consuming end with the lightest load (namely the consuming end with the least number of connected subject partitions) is selected according to the load level of the current consuming group, and the consuming end is informed that redundant partitions need to be connected.

When the primary copy of a theme partition goes offline, the consumption end connected with the primary copy randomly selects the primary copy of the theme partition to enter a polling waiting process (the polling interval can be set to be 1S), and waits for the primary copy of the theme partition to go online again or a new spare theme partition to go online. And if all the currently remaining theme partitions are in one-to-one correspondence with the consuming end, informing the consuming end to continue polling.

Example two

For simplicity, the parts of the device that are the same as those of the first embodiment are not repeated herein, and only the different parts of the second embodiment and the first embodiment are described.

In this embodiment, there are two storage modes of the theme message data at the server. One is the same file mode as the first embodiment, and the other is a memory mode. The file mode is suitable for storing message data with longer expiration time and larger volume, including logs, binlog of mysql, simple activity information of applications and the like.

The memory mode is suitable for pure fast message delivery, and these messages usually have short life cycle (perhaps only a few minutes), and require a relatively fast arrival time at the consuming end with a small transmission delay. This type of theme message data is not convenient to store using a file. Firstly, file storage is designed for messages with relatively long life cycles, and outdated data cannot be eliminated in time; secondly, for file storage, the data that can be consumed must be the subject message data that has landed (i.e. persisted to the hard disk with a valid logical version number), and there is a time difference between the subject message data being received and landed in the file storage mode, and the subject message data cannot be consumed in time.

The memory mode needs to be specified when creating a theme, and the mode of a theme cannot be changed once specified. A default expiration time may be specified when creating memory mode themes. If no message expiration time is specified when the message is produced, the message is eliminated by the default expiration time.

And storing the message data by depending on the memory in the memory mode, and mapping the theme zone to the memory for storage after receiving the theme message data generated by the production end. The memory usage of a theme zone needs to be specified (or default) when creating the theme. When allocating a theme partition, it is necessary to know the memory usage of the current server to avoid overflow.

Each topic partition is stored in memory in the form of a data paragraph skip table (see fig. 5). The data paragraph hopping table includes a plurality of data layers, each data layer including a data node. The number of nodes of the next data layer is greater than that of the nodes of the previous data layer, the next data layer comprises all the nodes of the previous data layer, and the data layer positioned at the bottom layer comprises all the nodes of the data paragraph skip table. Each node stores a paragraph data (Segment) of the subject message data of the subject partition, the node including its pointer data in the next data layer, and the pointer data of the next node of the data layer in which the node is located. The number of nodes spaced between the nodes of each layer is random. When the subject message data of the subject partition is indexed according to the logic version number of the subject message data, the top layer of the data paragraph jump table is searched downwards layer by layer, and the searching efficiency is greatly improved. For example, in fig. 5, the subject message data with the logical version number 117 needs to be searched:

1) comparing with the first node 21 of the top level (lever 3), finding the node with 117 greater than 21;

2) 117 is greater than 37 compared to the second node 37 at the top level, and 37 is the maximum value of the linked list at the level, then the search is started from the level below 37 (lever 2);

3) if 117 is greater than 71 and 71 is the maximum of the link list for that level, as compared to a node 71 following 37 in the second level, then the lookup starts at the level below 71 (lever 3);

4) if 117 is greater than 85, then look ahead, as compared to a node 85 behind 71 in the third level;

5) this node is found, 117 equals 117, compared to a node 117 following 85 in the third layer.

The specific embodiments described herein are merely illustrative of the spirit of the invention. Various modifications or additions may be made to the described embodiments or alternatives may be employed by those skilled in the art without departing from the spirit or ambit of the invention as defined in the appended claims.

Claims

1. A message data service cluster comprises a theme zone for storing theme message data, wherein the theme zone comprises different service terminals distributed in the service cluster; the method is characterized in that: the theme message data is stored in the service cluster in a memory mode or a file mode;

in the memory mode, the theme zone receives the theme message data generated by the production end and maps the theme message data into the memory of the service end;

in the file mode, the theme partition receives the theme message data generated by the production end and stores the theme message data in a hard disk of the server end;

the theme partition comprises a primary copy and a secondary copy distributed in different service terminals of the service cluster, the secondary copy is a redundant backup of the primary copy, the theme partition receives theme message data from the production terminal through the primary copy, and the theme partition responds to a consumption request of a consumption terminal through the primary copy;

the subject partition includes a set of kept synchronized replicas made up of the slave replicas, the slave replicas within the set of kept synchronized replicas being used for updated message data relative to other slave replicas; when the number of the slave copies in the synchronous copy keeping set is smaller than a preset minimum synchronous copy number, the subject partition does not receive subject message data from a production cluster;

under the condition of no guarantee level, after receiving the theme message data sent by the production end, the primary copy sends message submission success information to the production end;

under the rapid level, the master copy synchronizes to the slave copy after receiving the subject message data sent by the production end, and sends message submission success information to the production end after receiving the determined synchronization information of the first slave copy;

under the security level, the master copy synchronizes to the slave copy after receiving the subject message data sent by the production end, and sends message submission success information to the production end after receiving the determined synchronization information of the slave copies with the number larger than the preset value of the slave copies of the synchronization-maintaining copy set;

when the master copy goes offline, each slave copy in the set of kept synchronous copies reports the maximum logic version number to the management cluster, and the management cluster selects the slave copy corresponding to the maximum logic version number as the master copy according to the maximum logic version numbers of all the slave copies in the set of kept synchronous copies.

2. The message data service cluster of claim 1, wherein: the subject partition comprises a data passage, and the data passage comprises the subject message data and an index file; the index file records a one-to-one mapping relation between the logical version number of the subject message data and the physical offset of the subject message data.

3. A message data service cluster according to claim 2, characterized in that: the logic version number reflects the sequence of the theme message data reaching the service cluster.

4. A message data service cluster according to claim 3, characterized in that: in the memory mode, the subject partition includes a data paragraph skip table, and the data paragraph skip table includes a plurality of data layers; the number of nodes of the next data layer is greater than that of nodes of the previous data layer, the next data layer comprises all nodes of the previous data layer, and the data layer positioned at the bottom layer comprises all nodes of the data paragraph skip table; the node comprises paragraph data comprising the subject message data; the node includes its pointer data in the next data layer, pointer data of the next node of the data layer where the node is located.

5. The message data service cluster of claim 4, wherein: in the file mode, the subject partition records a range of physical offsets of the data paragraph and subject message data in the data paragraph.

6. The message data service cluster of claim 5, wherein: and the main copy records the effective logic version number of the latest received theme message data and the cache logic version number of the theme message data cached in the memory.

7. The message data service cluster of claim 6, wherein: and the production end polls the main copy of each theme partition so as to store the generated theme message data in each theme partition in a distributed manner.

8. The message data service cluster of claim 7, wherein: and the consumption end polls each theme zone to consume the theme message data.

9. The message data service cluster of claim 8, wherein: the topic partition allows at most one consuming side in the same consuming cluster to consume the topic message data at the same time.