CN111866072A - Message filter expansion method, application system and application method - Google Patents
Message filter expansion method, application system and application method Download PDFInfo
- Publication number
- CN111866072A CN111866072A CN202010523769.7A CN202010523769A CN111866072A CN 111866072 A CN111866072 A CN 111866072A CN 202010523769 A CN202010523769 A CN 202010523769A CN 111866072 A CN111866072 A CN 111866072A
- Authority
- CN
- China
- Prior art keywords
- message
- filter
- target
- filtering
- subscriber
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 40
- 238000001914 filtration Methods 0.000 claims abstract description 138
- 230000008569 process Effects 0.000 claims abstract description 11
- 238000012216 screening Methods 0.000 claims description 5
- 238000012217 deletion Methods 0.000 abstract description 6
- 230000037430 deletion Effects 0.000 abstract description 6
- 230000009286 beneficial effect Effects 0.000 abstract 1
- 238000012545 processing Methods 0.000 description 7
- 230000005540 biological transmission Effects 0.000 description 5
- 230000000903 blocking effect Effects 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 230000004048 modification Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004044 response Effects 0.000 description 3
- 239000002699 waste material Substances 0.000 description 3
- 238000004891 communication Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000002085 persistent effect Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
- H04L67/565—Conversion or adaptation of application format or content
- H04L67/5651—Reducing the amount or size of exchanged application data
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
- H04L67/568—Storing data temporarily at an intermediate stage, e.g. caching
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
- H04L67/568—Storing data temporarily at an intermediate stage, e.g. caching
- H04L67/5682—Policies or rules for updating, deleting or replacing the stored data
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The invention discloses an expanding method, an application system and an application method of a message filter, which relate to the field of general software and comprise the following steps: the message filter receives the filtering rule sent from the outside and stores the filtering rule to a cache cluster; the message filter is externally provided with a rule interface, receives an updating instruction sent from the outside through the rule interface, and dynamically updates the filtering rule stored in the cache cluster in real time according to the updating instruction. The invention has the beneficial effects that: a user can dynamically update and set the filtering rules in the cache cluster in real time through a rule interface according to the service requirements, so as to customize and expand the filtering rules; the method can efficiently process a large number of message subscription requests, realize dynamic addition and deletion of message filters and enhance the filtering function of the message filters.
Description
Technical Field
The invention relates to the field of general software, in particular to an expansion method, an application system and an application method of a message filter.
Background
In the management process of the telecommunication network, the system has massive messages to transmit among networks in a mode of publishing and subscribing, and the messages are timely and effectively filtered, so that invalid network transmission can be reduced, and the resource waste is reduced.
The message filtering is divided into client filtering and server filtering. The application client filtering method is easy to control and expand, but consumes large network resources. The application server side filtering method has less resource consumption, but common and well-known message middleware can only realize some simple filtering, for example, a subscriber uploads a simple filtering expression when subscribing, and dynamic modification of a filtering rule cannot be realized.
Disclosure of Invention
Aiming at the defects in the prior art, the invention aims to provide an expansion method, an application system and an application method of a message filter, wherein a user can dynamically update and set a filtering rule in a cache cluster in real time through a rule interface according to business requirements, so that the customized expansion filtering rule is realized, a large number of message subscription requests can be efficiently processed, a dynamic addition and deletion filter is realized, and the filtering function of the message filter is enhanced.
In order to achieve the above purposes, the technical scheme adopted by the invention is as follows:
an expansion method of a message filter, the expansion method comprising:
the message filter receives a filtering rule sent from the outside and stores the filtering rule to a cache cluster;
the message filter is externally provided with a rule interface, receives an updating instruction sent from the outside through the rule interface, and dynamically updates the filtering rule stored in the cache cluster in real time according to the updating instruction.
The invention also provides an application system of the message filter, which comprises:
the subscriber end is used for sending a message subscription request according to the instruction of the subscriber;
the message service module is used for feeding back a target message according to the message subscription request;
the cache cluster is used for storing the filtering rules and dynamically updating the stored filtering rules in real time according to the received updating instruction;
the message filter is used for acquiring the target message fed back by the message service module; the server is also used for acquiring the filtering rule from the cache cluster according to a preset strategy, filtering the target message according to the filtering rule and feeding back the processed target message to the subscriber end; and the message filter is provided with a regular interface, receives the filtering rule and the updating instruction sent by the outside through the regular interface and sends the filtering rule and the updating instruction to the cache cluster.
On the basis of the above technical solution, the message service module includes:
the message publisher end is used for receiving and outputting the message uploaded by the message publisher;
and the message service end is used for receiving the message subscription request sent by the message filter and the message sent by the message publisher end, acquiring the target message from the message according to the message subscription request and feeding the target message back to the message filter.
On the basis of the technical scheme, the message filter and the message service module are integrated in the same hardware terminal;
the message filters are stateless filters and support horizontal expansion, the number of the message filters is at least one, and each message filter is associated with one cache cluster.
On the basis of the technical scheme, the subscriber end sends a message subscription request to the message filter according to the instruction of the subscriber, and the message filter forwards the message subscription request to the message service module;
when the subscriber side is started, configuring a corresponding server side address for the subscriber side, automatically registering the message filter to the corresponding message service module according to the server side address after the message filter is started, wherein an address list formed by the addresses of all the message filters is stored in the message service module;
the subscriber acquires addresses of all current online message filters from the address list and selects an address of one online message filter according to a preset strategy, the subscriber end sends the message subscription request to the message filter corresponding to the address according to the instruction of the subscriber, the message filter corresponding to the address automatically forwards the message subscription request to the message service module corresponding to the server end address after receiving the message subscription request, the message service module corresponding to the server end address feeds the target message back to any message filter for filtering, and the message filter feeds the processed target message back to the subscriber end.
On the basis of the technical scheme, the subscriber end sends a message subscription request to the message service module according to the instruction of the subscriber;
and the message service module receives the message subscription request, feeds back the corresponding target message to any message filter for filtering, and feeds back the processed target message to the subscriber end.
On the basis of the technical scheme, after receiving the message sent by the message publisher end, the message server end persists the message into a message file, and the position and offset of each message in the message file are stored in an index file of the message file;
after receiving the message subscription request, the message server side inquires whether the message file has the corresponding target message, if so, the corresponding target message is obtained from the message file according to the index file and fed back to the message filter, and if not, the message subscription request is suspended until the message received by the message server side module contains the target message, and the corresponding target message is obtained from the message file according to the index file and fed back to the message filter.
On the basis of the technical scheme, the message subscription request comprises rule screening information, and the rule screening information comprises characteristic information of the subscriber;
the message uploaded by the message publisher comprises message content and message attributes;
and the message filter acquires the corresponding filtering rule from the cache module according to the rule screening information, wherein the filtering rule is used for judging whether the message attribute corresponding to each sub-target message in the target message to be processed meets a preset standard, deleting all the sub-target messages which do not meet the preset standard, and reserving all the sub-target messages which meet the preset standard as the processed target message.
On the basis of the above technical solution, the cache cluster is further configured to store the filtering rule uploaded by other terminals except the message filter.
The invention also provides an application method of the message filter, which is based on the application system of the message filter; the application method comprises the following steps:
the subscriber end sends a message subscription request according to the instruction of the subscriber;
the message service module feeds the target message back to the message filter according to the message subscription request;
The message filter acquires a target message corresponding to the message subscription request from a message service module, acquires a filtering rule from a cache cluster, and filters the target message according to the filtering rule to obtain the processed target message and feeds the processed target message back to the subscriber end;
in the above process, the filtering rule in the cache cluster is dynamically updated in real time, and the real-time dynamic updating specifically includes the following steps:
and receiving the filtering rule sent from the outside through the rule interface arranged on the message filter and storing the filtering rule to the cache cluster, and receiving the updating instruction sent from the outside through the rule interface arranged on the message filter to dynamically update the filtering rule stored in the cache cluster in real time.
Compared with the prior art, the invention has the advantages that:
(1) the filtering rules are maintained by the cache cluster, the message filters acquire the required filtering rules from the cache cluster, the consistency of data is ensured by the cluster, the complexity of the message filters is reduced, the message filters are stateless and easy to realize horizontal extension, a plurality of message filters can be arranged when a large number of message subscription requests are processed, the large number of message subscription requests can be efficiently processed, and therefore the filtering function of the message filters is enhanced.
(2) The method has the advantages that the application data cannot be influenced by randomly starting and closing one message filter instance in a filter cluster formed by a plurality of message filters, dynamic addition and deletion of the filters are realized, the filtering efficiency is improved, and idle waste of the message filters is avoided.
(3) By setting the rule interface on the message filter, a user can dynamically update and set the filtering rule in the cache cluster in real time through the rule interface according to the service requirement, so as to customize and expand the filtering rule.
Drawings
FIG. 1 is a functional block diagram of an application system of a message filter according to an embodiment of the present invention;
fig. 2 is a schematic diagram of a message subscription transmission flow of an application system of a message filter according to an embodiment of the present invention.
Fig. 3 is a schematic diagram of a subscriber-side thread model according to an embodiment of the present invention.
FIG. 4 is a timing diagram illustrating the interaction of a message filter with other components in an embodiment of the present invention.
In the figure: 1-subscriber end, 2-message filter, 3-cache cluster, 4-message service module, 41-message publisher end, and 42-message service end.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples.
As shown in fig. 1 and fig. 2, an embodiment of the present invention provides an extension method, an application system, and an application method of a message filter. The message filter 2 provides a rule interface for external setting, and the message filter 2 stores the filtering rule in the cache cluster 3 after receiving the filtering rule through the rule interface. The user or other application program dynamically modifies the stored filtering rules in the cache cluster 3 through the rule interface. The user or other application programs can set the filtering rules through a remote calling rule interface, such as a common restful interface and a webservice interface. When the expansion method of the message filter 2 is applied to the message subscription transmission process, after a subscriber issues a message subscription request, the message storage side feeds back a corresponding target message, the message filter 2 acquires a filtering rule from the cache cluster 3 according to a preset strategy and filters the target message according to the filtering rule to obtain a processed target message, and feeds back the processed target message to the subscriber terminal 1 sending the message subscription request to complete the message subscription transmission process. The integration of the message filter 2 with the message server module 4 may enhance the message filtering capabilities of the message service module 4.
The filtering rules are maintained by the cache cluster 3, the message filter 2 acquires the required filtering rules from the cache cluster 3, the consistency of data is ensured by the cluster, the complexity of the message filter 2 is reduced, the message filter is stateless and easy to realize horizontal extension, a plurality of message filters 2 can be arranged when a large number of message subscription requests are processed, the large number of message subscription requests can be efficiently processed, and therefore the filtering function of the message filter 2 is enhanced.
The random starting and closing of one message filter 2 instance in a filter cluster formed by a plurality of message filters 2 can not affect the application data, thereby realizing dynamic addition and deletion of the filters, improving the filtering efficiency and avoiding idle waste of the message filters 2.
By setting the rule interface on the message filter 2, the user can dynamically update and set the filtering rule in the cache cluster 3 in real time through the rule interface according to the service requirement, thereby realizing the customization and extension of the filtering rule.
Example one
As shown in fig. 1, the present embodiment provides an extension method of a message filter, where the extension method includes:
the message filter 2 receives the filtering rules sent from outside and stores the filtering rules in a cache cluster 3.
The message filter 2 sets a rule interface to the outside, receives an update instruction sent from the outside through the rule interface, and dynamically updates the filter rule stored in the cache cluster 3 in real time according to the update instruction.
In this embodiment, the message filter 2 provides a rule interface for external application, and the filter receives the filtering rule and stores the filtering rule in the cache cluster 3. The user or other application dynamically modifies the filtering rules through this interface.
Further, the cache cluster 3 may further store the filtering rule uploaded by other terminals except the message filter 2. The update instructions received through the rule interface may dynamically update all the filter rules already stored in the cache cluster 3 in real time.
Example two
As shown in fig. 1 to 4, the present embodiment provides an application system of a message filter, where the application system includes:
the subscriber terminal 1, the subscriber terminal 1 sends a message subscription request according to the instruction of the subscriber.
And the message filter 2 is connected with the subscriber terminal 1, and is configured to obtain a target message corresponding to the message subscription request from the message service module 4, where the target message is not yet subjected to filtering processing, and also obtain a filtering rule from the cache cluster 3, and filter the target message according to the filtering rule to obtain the processed target message and feed the processed target message back to the subscriber terminal 1. The message filter 2 has a rule interface, and receives a filtering rule and an update instruction through the rule interface and sends the filtering rule and the update instruction to the cache cluster 3.
The cache cluster 3 is connected to the message filter 2, and is configured to store the filtering rule, dynamically update the stored filtering rule in real time according to the update instruction, and send the internally stored filtering rule to the message filter 2 for filtering. The message filter 2 receives the filtering rule and stores the filtering rule in the cache cluster 3. The user or other application dynamically modifies the filter rules through the rule interface.
The message service module 4 is connected to the subscriber terminal 1 and the message filter 2, and configured to feed back the target message to the message filter 2 according to the message subscription request. The message service module 4 includes: and a message publisher end 41 through which the message publisher uploads the message. And a message server 42, connected to the message publisher 41, configured to receive the message subscription request sent by the message filter 2 and the message sent by the message publisher 41, and obtain the target message from the message according to the message subscription request and feed the target message back to the message filter 2.
In this embodiment, the message subscription request sent by the subscriber through the subscriber terminal 1 may be sent to the message service module 4 through forwarding of the message filter 2, or may be sent directly to the message service module 4. As shown in fig. 3, a new message subscription request is added to the blocking queue, the message subscription request is cyclically taken out from the blocking queue, then the message subscription request is sent to the message server 42 or the message filter 2, and the response result (including the target message) of the message server 42 is processed, where processing the response result includes sending the target message to the work thread pool, and filtering the target message by the thread corresponding to the filtering process to obtain the processed target message, and in this process, a next message subscription request may be created and delivered to the blocking queue.
The filtering processing comprises that the message service module 4 feeds back a target message to be processed to the message filter 2 according to the received message subscription request, the message filter 2 obtains a filtering rule from the cache cluster 3 according to a preset strategy, filters the target message to be processed to obtain a processed target message, and sends the processed target message to the subscriber terminal 1.
The message subscription request includes rule filtering information, and the rule filtering information includes characteristic information of the subscriber, such as a subscriber ID. The message uploaded by the message publisher comprises message content and message attributes. Each message has corresponding message content and message attributes.
The message server 42 receives the message sent by the message publisher 41, and then persists the message into a message file, where the position and offset of each message in the message file are stored in an index file of the message file. After receiving the message subscription request, the message server 42 queries whether the message file has the corresponding target message, and if so, obtains the corresponding target message from the message file according to the index file and feeds the target message back to the message filter 2; if the message does not exist, the message subscription request is suspended until the message server 42 receives the message sent by the message publisher 41 again and persists the message to the message file. Then, whether the corresponding target message exists in the message file is further queried, if so, the corresponding target message is obtained from the message file according to the index file and fed back to the message filter 2, and if not, the message subscription request is suspended until the target message exists in the messages received by the message server 42.
After the message filter 2 receives the target message to be processed fed back by the message service module 4 according to the message subscription request, processing the rule screening information associated with the message subscription request according to a preset policy, so as to obtain the corresponding filtering rule from the cache module, for example, the preset policy is to query the corresponding rule from the cache cluster 3 according to the subscriber ID, the filtering rule is to determine whether the message attribute corresponding to each sub-target message (the target message is composed of at least one sub-target message, each sub-target message also has corresponding message content and message attribute) in the target message to be processed meets a preset standard, delete all the sub-target messages whose message attributes do not meet the preset standard in the target message to be processed, and reserving all the sub-target messages of which the message attributes meet the preset standard in the target message to be processed as the processed target message.
EXAMPLE III
Based on the above embodiment, the message filter 2 and the message service module 4 are integrated in the same terminal. The message filter 2 and the message service end 42 are deployed on the same machine, and the request data packet does not pass through a physical network device, so that the communication overhead is reduced.
In this embodiment, the message filters 2 are stateless filters and support horizontal expansion, the number of the message filters 2 is at least one, and each message filter 2 is associated with one cache cluster 3. The message filter 2 is stateless, and no data is stored in the process, so that any addition or deletion of the message filter 2 instance does not affect data transmission. When the concurrency of the instant message subscription requests is large, the pressure of a single message filter 2 instance can be relieved by starting a plurality of message filter 2 instances.
Example four
Based on the first to third embodiments, as shown in fig. 2 and fig. 4, the subscriber sends a message subscription request to the message filter 2 through the subscriber terminal 1, and the message filter 2 forwards the message subscription request to the message service module 4.
When the subscriber terminal 1 is set, a corresponding server address is configured for the subscriber terminal 1, the message filter 2 is automatically registered to the corresponding message service module 4 according to the server address after being started, and the message service module 4 stores all address lists formed by the addresses of the message filter 2.
The message filter 2 and the message service 42 may be deployed on the same machine. The user needs to configure the address of the message server 42 hung on the message filter 2, that is, the user needs to specify the address of the message server 42 when creating the subscriber terminal 1 using the API. The message filter 2 will automatically register with the message server 42 when it is started and the server will remain active through the heartbeat. The subscriber end 1 can automatically recognize the address of the message filter 2 after starting, and circularly sends the message subscription request. After receiving the message subscription request, the message filter 2 directly forwards the message to the message server 42.
When the message subscription request concurrency is large, the pressure of a single message filter 2 instance can be relieved by starting multiple message filter 2 instances. At this time, the subscriber may obtain the address list of all current online message filter 2 instances from the message server 42, and then select one message filter 2 address to send the message subscription request and the target message to be processed according to a policy (e.g., polling, random policy), after receiving the message subscription request, the message filter 2 directly forwards the message subscription request to the corresponding message server 42, and feeds the target message back to any of the message filters 2 for the above filtering process to the message server 42, and the message filter 2 sends the processed target message to the subscriber terminal 1.
In this embodiment, after the subscriber uploads the message subscription request through the subscriber terminal 1, the subscriber terminal 1 forwards the message subscription request to the message service module 4, the message service module 4 feeds back the corresponding target message to be processed to the message filter 2, the message filter 2 obtains the filtering rule from the buffer module to filter the target message to be processed to obtain the processed target message, and sends the processed target message to the subscriber terminal 1.
EXAMPLE five
Based on the first to third embodiments, the subscriber sends a message subscription request to the message service module 4 through the subscriber terminal 1.
When the subscriber terminal 1 is set, a corresponding server address is configured for the subscriber terminal 1, the message filter 2 is automatically registered to the corresponding message service module 4 according to the server address after being started, and the message service module 4 stores all address lists formed by the addresses of the message filter 2.
The message filter 2 and the message service 42 may be deployed on the same machine. The user needs to configure the address of the message server 42 hung on the message filter 2, that is, the user needs to specify the address of the message server 42 when creating the subscriber terminal 1 using the API. The message filter 2 will automatically register with the message server 42 when it is started and the server will remain active through the heartbeat.
After a subscriber uploads a message subscription request through the subscriber terminal 1, the subscriber terminal 1 directly sends the message subscription request to the message service module 4, the message service module 4 feeds back a corresponding target message to be processed to the message filter 2, the message filter 2 obtains a filtering rule from the cache module to filter the target message to be processed to obtain a processed target message, and sends the processed target message to the subscriber terminal 1.
When the message subscription request concurrency is large, the pressure of a single message filter 2 instance can be relieved by starting multiple message filter 2 instances. At this time, the subscriber may obtain the address list of all current online message filter 2 instances from the message service end 42, then select one message filter 2 address according to a policy (e.g. polling, random policy) to send the target message to be processed, and the message filter 2 performs filtering processing after receiving the target message to be processed.
EXAMPLE six
As shown in fig. 2 and fig. 4, an application method of a message filter supports dynamic setting of filtering rules based on an application system of the message filter 2, the message filter 2 can support message filtering at a message server 42 by being integrated at the message server 42, and the message filter 2 supports horizontal extension. The application method comprises the following steps:
the subscriber sends a message subscription request through the subscriber terminal 1. The message service module 4 feeds the target message back to the message filter 2 according to the message subscription request.
The message filter 2 obtains a target message corresponding to the message subscription request from the message service module 4, obtains a filtering rule from the cache cluster 3, and performs filtering processing on the target message according to the filtering rule to obtain a processed target message, and feeds the processed target message back to the subscriber terminal 1. The message filter 2 has a rule interface, and receives a filtering rule and an update instruction through the rule interface and sends the filtering rule and the update instruction to the cache cluster 3.
The cache cluster 3 is connected to the message filter 2, stores the filtering rules, dynamically updates the stored filtering rules in real time according to the update instruction, and is further configured to send the internally stored filtering rules to the message filter 2 for filtering. The cache cluster 3 is further configured to store the filtering rule uploaded by other terminals except the message filter 2.
The message service module 4 includes: and a message publisher end 41 through which the message publisher uploads the message. And a message server 42, connected to the message publisher 41, configured to receive the message subscription request sent by the message filter 2 and the message sent by the message publisher 41, and obtain the target message from the message according to the message subscription request and feed the target message back to the message filter 2.
In this embodiment, the message filter 2 provides a rule interface, such as an RPC interface, to the outside, and the message filter 2 stores the filtering rule in the cache cluster 3 after receiving the filtering rule. The user or other application dynamically modifies the filter rules through this rule interface. The user sets a filtering rule to the message filter 2 through a remote calling rule interface, the message filter 2 stores the filtering rule in the cache cluster 3 after receiving the filtering rule, and the rule data is not maintained in the process of the message filter 2. When the message filter 2 needs to filter the target message, the corresponding filtering rule is temporarily inquired from the caching cluster 3 according to the subscriber ID. The filtering rules define the rules that the Subscriber needs to satisfy to receive the message, such as Subscriber1: [1001.. 1010], Subscriber2: [1011.. 1020], Subscriber3: [1021.. 1030 ]. I.e., subscribers with ID subscribe 1 only consume messages with device IDs (message attributes) within 1001 through 1010. The filtering rules are maintained by the cache cluster 3, which reduces the complexity of the filter, makes it stateless and easily realizes the lateral extension. Filtering according to the business logic can be freely performed. For example, alarm messages with device IDs of 1000 to 2000 may also be filtered out
The message publisher sends the message to the message service 42. The message publishing process is the same as that of other common message middleware. Besides the body, the message body also comprises some message attributes customized by the user, such as the ID of the group to which the message belongs, the ID of the device sending the message, and the like. The message is sent to a queue under the specified subject of the message service 42. When a user sends a message by using the interface, relevant attributes can be set for the message, so that the message can be filtered in the message filter 2 according to the message attributes. The location of the message in the message file and the offset of the current message are stored in an index file. When the message server 42 receives the pull message request, the position of the message in the message file is found from the index file according to the offset of the request message, and then the message is obtained from the message file. For example, a message attribute "DeviceId 1003" is set. The user needs to configure the message filter 2 with the address of his external message service 42. The message filter 2 registers with the message service 42 when it is started and the message service 42 remains active through a heartbeat.
The subscriber pulls the target message to be processed from the message server 42 through the message filter 2. After the message filter 2 is activated, the subscriber can perceive the existence of the message filter 2, and a request for pulling a message (i.e., a message subscription request) from the subscriber is sent to the message filter 2 and then forwarded to the message server 42. If the message filter 2 is not deployed, the subscriber directly sends the request to the server. The message subscription requests which are continuously created form a blocking queue, the message subscription requests are circularly taken out from the blocking queue and sent to the message filter 2 or the message service terminal 42, and after the message service terminal 42 returns the target message, the target message is filtered by the working thread of the message filtering processing and the processed target message is fed back to the subscriber terminal 1.
The message filter 2 is stateless, and no data is stored in the process, so that any addition or deletion of the message filter 2 instance does not affect the data. When there is a large amount of concurrency of pull message requests, the stress of a single instance can be alleviated by starting multiple instances of the message filter 2. Wherein, after the message filter 2 is started, it actively registers with the configured message server 42 without user participation. There is no identity between the subscriber end 1 and the message filter 2, and the subscriber end 1 directly establishes a TCP connection after acquiring the IP port monitored by the message filter 2. The subscriber obtains a list of addresses of all current online message filter 2 instances from the message service 42 and then selects a message filter 2 address to send a pull message request according to a policy (e.g., polling, random policy).
After receiving the pull request, the message filter 2 directly forwards the pull request to the corresponding message server 42. When a user creates the subscriber terminal 1 by using the API, the address of the message server 42, the topic of the subscription, the offset of the initial message, and the subscriber ID need to be specified. The initial offset is set for subscription, for example, the message server 42 stores 1 to 1000 messages, and if subscription specifies to pull from 101, a total of 900 messages will be pulled back. When the subscriber terminal 1 tries to pull the 1001 st message, it is blocked by the message service terminal 42 until there is the latest message. The subscriber can automatically recognize the address of the message filter 2 after starting and send the pull message request circularly. The message filter 2 receives the pull message request and then directly forwards the pull message request to the message server 42.
The message server 42 returns the pending target message meeting the requirements to the message filter 2. After receiving the message subscription request, the message server 42 finds that there is a new subscribed message, and immediately returns the new subscribed message to the filter. If not, the request is suspended until there is a new message. Wherein no user involvement is required. After receiving the request of pulling the message, the message server 42 queries whether there is a message meeting the condition from the persistent message file, if so, immediately returns the message meeting the requirement as a target message to be processed, if not, suspends the request, and after receiving a new message issued by the publisher again, the message server 42 continuously checks whether the message subscription request meets the suspended request until there is a new message meeting the condition, and then gives a response to the request, and the long polling mechanism ensures that the message subscription request is not too frequent.
And the message filter 2 filters the messages according to the filtering rules, and returns the messages which accord with the filtering rules in the target messages to be processed to the subscribers as the processed target messages. After receiving the target message returned by the message server 42, the message filter 2 checks whether the target message satisfies the filtering rule, discards the message that does not satisfy the filtering rule, and returns the message that satisfies the filtering rule to the subscriber without user participation. For example, the filtering rule corresponding to the subscriber can be found from the cache according to the subscriber ID in the pull message request, and then whether the message meets the filtering rule is judged and processed. For example, if the rule of subscriber substriber 1 is [1001 … 1010], then the message with attribute "DeviceId 1003" meets the filtering condition and is returned to the subscriber.
The present invention is not limited to the above-described embodiments, and it will be apparent to those skilled in the art that various modifications and improvements can be made without departing from the principle of the present invention, and such modifications and improvements are also considered to be within the scope of the present invention. Those not described in detail in this specification are within the skill of the art.
Claims (10)
1. An expansion method of a message filter, the expansion method comprising:
the message filter receives a filtering rule sent from the outside and stores the filtering rule to a cache cluster;
the message filter is externally provided with a rule interface, receives an updating instruction sent from the outside through the rule interface, and dynamically updates the filtering rule stored in the cache cluster in real time according to the updating instruction.
2. An application of a message filter, the application comprising:
the subscriber end is used for sending a message subscription request according to the instruction of the subscriber;
the message service module is used for feeding back a target message according to the message subscription request;
the cache cluster is used for storing the filtering rules and dynamically updating the stored filtering rules in real time according to the received updating instruction;
The message filter is used for acquiring the target message fed back by the message service module; the server is also used for acquiring the filtering rule from the cache cluster according to a preset strategy, filtering the target message according to the filtering rule and feeding back the processed target message to the subscriber end; and the message filter is provided with a regular interface, receives the filtering rule and the updating instruction sent by the outside through the regular interface and sends the filtering rule and the updating instruction to the cache cluster.
3. The message filter application of claim 2 wherein the message service module comprises:
the message publisher end is used for receiving and outputting the message uploaded by the message publisher;
and the message service end is used for receiving the message subscription request sent by the message filter and the message sent by the message publisher end, acquiring the target message from the message according to the message subscription request and feeding the target message back to the message filter.
4. The message filter application of claim 2 wherein the message filter and the message service module are integrated in the same hardware terminal;
the message filters are stateless filters and support horizontal expansion, the number of the message filters is at least one, and each message filter is associated with one cache cluster.
5. The application system of the message filter as claimed in claim 2, wherein the subscriber end sends a message subscription request to the message filter according to the subscriber's instruction, and the message filter forwards the message subscription request to the message service module;
when the subscriber side is started, configuring a corresponding server side address for the subscriber side, automatically registering the message filter to the corresponding message service module according to the server side address after the message filter is started, wherein an address list formed by the addresses of all the message filters is stored in the message service module;
the subscriber acquires addresses of all current online message filters from the address list and selects an address of one online message filter according to a preset strategy, the subscriber end sends the message subscription request to the message filter corresponding to the address according to the instruction of the subscriber, the message filter corresponding to the address automatically forwards the message subscription request to the message service module corresponding to the server end address after receiving the message subscription request, the message service module corresponding to the server end address feeds the target message back to any message filter for filtering, and the message filter feeds the processed target message back to the subscriber end.
6. The message filter application system of claim 2, wherein the subscriber side sends a message subscription request to the message service module according to the subscriber's instruction;
and the message service module receives the message subscription request, feeds back the corresponding target message to any message filter for filtering, and feeds back the processed target message to the subscriber end.
7. The message filter application system of claim 3, wherein the message server side persists the message into a message file after receiving the message sent by the message publisher side, and the position and offset of each message in the message file are stored in an index file of the message file;
after receiving the message subscription request, the message server side inquires whether the message file has the corresponding target message, if so, the corresponding target message is obtained from the message file according to the index file and fed back to the message filter, and if not, the message subscription request is suspended until the message received by the message server side module contains the target message, and the corresponding target message is obtained from the message file according to the index file and fed back to the message filter.
8. The message filter application system of claim 3, wherein the message subscription request includes rule filtering information including characteristic information of the subscriber;
the message uploaded by the message publisher comprises message content and message attributes;
and the message filter acquires the corresponding filtering rule from the cache module according to the rule screening information, wherein the filtering rule is used for judging whether the message attribute corresponding to each sub-target message in the target message to be processed meets a preset standard, deleting all the sub-target messages which do not meet the preset standard, and reserving all the sub-target messages which meet the preset standard as the processed target message.
9. The message filter application system of claim 2, wherein the cache cluster is further configured to store the filtering rules uploaded by other terminals besides the message filter.
10. A message filter application method, characterized in that, based on the message filter application system of any one of the above claims 2-9; the application method comprises the following steps:
the subscriber end sends a message subscription request according to the instruction of the subscriber;
The message service module feeds the target message back to the message filter according to the message subscription request;
the message filter acquires a target message corresponding to the message subscription request from a message service module, acquires a filtering rule from a cache cluster, and filters the target message according to the filtering rule to obtain the processed target message and feeds the processed target message back to the subscriber end;
in the above process, the filtering rule in the cache cluster is dynamically updated in real time, and the real-time dynamic updating specifically includes the following steps:
and receiving the filtering rule sent from the outside through the rule interface arranged on the message filter and storing the filtering rule to the cache cluster, and receiving the updating instruction sent from the outside through the rule interface arranged on the message filter to dynamically update the filtering rule stored in the cache cluster in real time.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010523769.7A CN111866072A (en) | 2020-06-10 | 2020-06-10 | Message filter expansion method, application system and application method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010523769.7A CN111866072A (en) | 2020-06-10 | 2020-06-10 | Message filter expansion method, application system and application method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111866072A true CN111866072A (en) | 2020-10-30 |
Family
ID=72987456
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010523769.7A Pending CN111866072A (en) | 2020-06-10 | 2020-06-10 | Message filter expansion method, application system and application method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111866072A (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112702311A (en) * | 2020-11-30 | 2021-04-23 | 锐捷网络股份有限公司 | Port-based message filtering method and device |
| CN112738245A (en) * | 2020-12-28 | 2021-04-30 | 青岛海尔科技有限公司 | Method and device for determining equipment response information, storage medium and electronic device |
| CN112787993A (en) * | 2020-12-25 | 2021-05-11 | 北京金万维科技有限公司 | High-concurrency HTTP request caching and content pushing system and method based on UDP |
| CN113079087A (en) * | 2021-03-31 | 2021-07-06 | 上海天旦网络科技发展有限公司 | Interconnected data gateway, and data processing system and method based on interconnected data gateway |
| CN115914393A (en) * | 2022-11-29 | 2023-04-04 | 北京科银京成技术有限公司 | Communication method, device, equipment and medium |
| CN116308830A (en) * | 2023-05-25 | 2023-06-23 | 凯泰铭科技（北京）有限公司 | Rule management and execution release processing system based on insurance finance |
| CN117397225A (en) * | 2021-03-29 | 2024-01-12 | 谷歌有限责任公司 | Distributed messaging service based on asynchronous events |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20030120785A1 (en) * | 2001-12-20 | 2003-06-26 | International Business Machines Corporation | Message filtering |
| US20040236780A1 (en) * | 2003-02-25 | 2004-11-25 | Michael Blevins | Systems and methods for client-side filtering of subscribed messages |
| US20050132008A1 (en) * | 2003-12-10 | 2005-06-16 | International Business Machines Corporation | Database supported message routing |
| US20050147221A1 (en) * | 2003-12-30 | 2005-07-07 | Aoki Norihiro E. | Method and apparatus for managing subscription-type messages |
| CN101364952A (en) * | 2007-08-08 | 2009-02-11 | 华为技术有限公司 | Transaction subscribing method, filtering rule configuring method and related equipment |
| CN110956485A (en) * | 2018-09-27 | 2020-04-03 | 千寻位置网络有限公司 | Message subscription method and component based on RocktMQ and message subscription system |
-
2020
- 2020-06-10 CN CN202010523769.7A patent/CN111866072A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20030120785A1 (en) * | 2001-12-20 | 2003-06-26 | International Business Machines Corporation | Message filtering |
| US20040236780A1 (en) * | 2003-02-25 | 2004-11-25 | Michael Blevins | Systems and methods for client-side filtering of subscribed messages |
| US20050132008A1 (en) * | 2003-12-10 | 2005-06-16 | International Business Machines Corporation | Database supported message routing |
| US20050147221A1 (en) * | 2003-12-30 | 2005-07-07 | Aoki Norihiro E. | Method and apparatus for managing subscription-type messages |
| CN101364952A (en) * | 2007-08-08 | 2009-02-11 | 华为技术有限公司 | Transaction subscribing method, filtering rule configuring method and related equipment |
| CN110956485A (en) * | 2018-09-27 | 2020-04-03 | 千寻位置网络有限公司 | Message subscription method and component based on RocktMQ and message subscription system |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112702311A (en) * | 2020-11-30 | 2021-04-23 | 锐捷网络股份有限公司 | Port-based message filtering method and device |
| CN112702311B (en) * | 2020-11-30 | 2022-10-14 | 锐捷网络股份有限公司 | Port-based message filtering method and device |
| CN112787993A (en) * | 2020-12-25 | 2021-05-11 | 北京金万维科技有限公司 | High-concurrency HTTP request caching and content pushing system and method based on UDP |
| CN112738245A (en) * | 2020-12-28 | 2021-04-30 | 青岛海尔科技有限公司 | Method and device for determining equipment response information, storage medium and electronic device |
| CN112738245B (en) * | 2020-12-28 | 2023-03-28 | 青岛海尔科技有限公司 | Method and device for determining equipment response information, storage medium and electronic device |
| CN117397225A (en) * | 2021-03-29 | 2024-01-12 | 谷歌有限责任公司 | Distributed messaging service based on asynchronous events |
| CN113079087A (en) * | 2021-03-31 | 2021-07-06 | 上海天旦网络科技发展有限公司 | Interconnected data gateway, and data processing system and method based on interconnected data gateway |
| CN113079087B (en) * | 2021-03-31 | 2022-11-22 | 上海天旦网络科技发展有限公司 | Interconnected data gateway, and data processing system and method based on interconnected data gateway |
| CN115914393A (en) * | 2022-11-29 | 2023-04-04 | 北京科银京成技术有限公司 | Communication method, device, equipment and medium |
| CN116308830A (en) * | 2023-05-25 | 2023-06-23 | 凯泰铭科技（北京）有限公司 | Rule management and execution release processing system based on insurance finance |
| CN116308830B (en) * | 2023-05-25 | 2023-08-18 | 凯泰铭科技（北京）有限公司 | Rule management and execution release processing system based on insurance finance |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111866072A (en) | Message filter expansion method, application system and application method | |
| CN111131501B (en) | MQTT protocol-based message pushing system and method | |
| US8769025B2 (en) | Cluster server of an instant messaging system and messaging method between clusters | |
| CN102377686B (en) | Message subscription system, method and device | |
| CN109274730B (en) | Internet of things system, MQTT message transmission optimization method and device | |
| EP2321908B1 (en) | Method and system for message processing | |
| KR100817098B1 (en) | Information exchange system, management server, and method for reducing network load used in the same | |
| US6850968B1 (en) | Reduction of network server loading | |
| CN110971686B (en) | Electronic mall customer service system supporting high concurrency and high availability | |
| KR100825349B1 (en) | Information exchange system and management server, terminal unit, and method for reducing network load used in the same | |
| CN108768826A (en) | Based on the message route method under MQTT and Kafka high concurrent scenes | |
| CN102045270A (en) | Chat server system, method for constructing chat server system and chat system | |
| US6718369B1 (en) | Method, system and program for efficiently distributing serial electronic publications | |
| CN104243611A (en) | Distribution thought based message service middleware system | |
| CN111031113B (en) | User queuing method for supporting platform-level customer service system | |
| CN112583895B (en) | TCP communication method, system and device | |
| CN110380967B (en) | SSE technology-based server message pushing method | |
| CN111475315A (en) | Server and subscription notification push control and execution method | |
| CN108737161B (en) | Real-time data bus management method and system for intelligent agent cooperation system | |
| CN110868323A (en) | Bandwidth control method, device, equipment and medium | |
| CN102209084A (en) | Processing method of Push message, apparatus thereof and communication system | |
| CN109428745B (en) | JMS message forwarding method, device, electronic equipment and storage medium | |
| CN109063140A (en) | A kind of data query method, transfer server and computer readable storage medium | |
| US9374328B1 (en) | Selective messaging using online presence information | |
| KR101272077B1 (en) | System and method for providing push service |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20201030 |
|
| RJ01 | Rejection of invention patent application after publication |