[WIP][ISSUE #4043]initial implementation of filter and transform #4365

pmupkin · 2023-08-15T14:13:39Z

Initial Informal Implementation of Filter and Transform. Link to relevant issues: #825 . #4043 . #3263

1. Overview

Filter and transformer are components of EventMesh used for event filtering and event transformation. During the process of sending and receiving events in EventMesh, rules for event filtering and transformation can be configured to achieve the capability of filtering and transforming events.

Filter refers to the process of filtering events in event subscriptions using configured filtering rules, and then routing the filtered events to event destinations.

Transformer refers to the process of transforming the content of events by configuring types in event subscriptions, converting CloudEvents standard events into event types that event destinations can accept.

Both of these functionalities are implemented using Jackson and JsonPath.

Filter

Filter is an event filtering module. Its primary function is to match events generated by the event source with filtering rules. Only when a successful match occurs, the event will be routed to the event destination associated with the filtering rule. The filtering rule must have the same structure as the matched event.

Supported filtering conditions:
- Prefix Match: Specifies matching based on the value of a particular field with a prefix. The keyword is prefix.
- Anything-But Match: Specifies matching a string-type field with any value except the ones provided. The keyword is anything-but.
- Suffix Match: Specifies matching based on the value of a particular field with a suffix. The keyword is suffix.
- Numeric Match: Specifies matching within a range of numeric values. You need to specify an operator and a number, with operators including ">" (greater than), ">=" (greater than or equal to), "<=" (less than or equal to), "<" (less than), "=" (equal to), and "!=" (not equal to). Numeric values are treated as Doubles by default and should be stored in an array format. The keyword is numeric.
- Existence Match: Checks whether a field exists, with options to set it as true if it exists or false if it doesn't. The keyword is exists.
- Specified Match: Determines whether a value exists in a particular field. In this type of matching condition, there's no need to specify a value.
- Composite Match: In addition to individual matching rules, Filter also supports combining multiple rules to create more complex event patterns.
Matching between different fields is performed using AND logic, while matching within the same field is performed using OR logic by default.
Processing Flow

The implementation of the filter primarily relies on patterns and conditions. The execution flow of the filter is illustrated in the diagram below:

PatternBuilder: Responsible for parsing the condition content and generating patterns.

Pattern: Evaluates events based on filtering conditions and returns results.

ConditionBuilder: Constructs specific conditions based on different condition keywords.

Condition: Includes various matching conditions, each with its own matching implementation.

Usage Example

String condition = "{
    "source":[
        {
            "prefix":"eventmesh."
        },
				{
            "suffix":"eventmesh."
        },
    ]
}";
String event = "{
  "id": "7bf73129-1428-4cd3-a780-95db273d1602",
  "source": "eventmesh.source",
  "time": "2023-11-11T21:29:54Z",
  "data":{
        "name":"filter-test",
        "example":"demo"
    }
}";
Pattern pattern =  PatternBuilder.build(condition);
Boolean res = pattern.filter(cont);

Transformer

The transformer is an event conversion module designed to transform CloudEvents standard events into types that the event target can process.

Supported Types

Transformer supports three types of transformations:

Direct Transfer: No event transformation is performed; the CloudEvents standard event is directly routed to the target.

Constant: Regardless of the event content, the event triggers the event target without sending the event content to the target.

Variable: Extract variable values from the CloudEvents standard event and route them to the event target according to the format defined in the template. Users can define custom templates and specify the list of variables to extract. The transformer will extract the variables from the event and route them to the event target according to the template's defined format.

Processing Flow

As an example of template matching in the transformer, the implementation is as follows:

Usage Example

//ORIGIN
Transformer transformer =new TransformerBuilder.Builder(TransformerType.ORIGINAL).build();
String output = transformer.transform(EVENT);
//Constant
Transformer transformer = new TransformerBuilder.Builder(TransformerType.CONSTANT).setContent("constant test").build();
String output = transformer.transform(EVENT);

github-actions

Welcome to the Apache EventMesh community!!
This is your first PR in our project. We're very excited to have you onboard contributing. Your contributions are greatly appreciated!

Please make sure that the changes are covered by tests.
We will be here shortly.
Let us know if you need any help!

Want to get closer to the community?

WeChat Assistant	WeChat Public Account	Slack
		Join Slack Chat

Mailing Lists:

Name	Description	Subscribe	Unsubscribe	Archive
Users	User support and questions mailing list	Subscribe	Unsubscribe	Mail Archives
Development	Development related discussions	Subscribe	Unsubscribe	Mail Archives
Commits	All commits to repositories	Subscribe	Unsubscribe	Mail Archives
Issues	Issues or PRs comments and reviews	Subscribe	Unsubscribe	Mail Archives

mxsm

@pmupkin Please link this pr to related issue

pandaapo · 2023-08-16T00:20:24Z

First of all, I like your way of submitting PR: breaking down a task into multiple PRs, which avoids submitting a very large PR all at once. Your approach can greatly improve the quality of review and reduce the difficulty of review. 👍

Secondly, since your initial implementation has not been applied to the business of EventMesh yet, could you please add documentation to your code (including classes and methods) to help developers and reviewers understand their purpose and future roles in EventMesh?

Also, please refer to https://eventmesh.apache.org/community/contribute/contribute/#code-style to check your code style.

pandaapo · 2023-08-16T00:23:24Z

eventmesh-common/src/main/java/org/apache/eventmesh/common/filter/FilterType.java

+ + " \"f-count\": [{\"exists\": false}]\n" + " }\n" + " },\n"
+ + " \"prefix\": [{\"prefix\": \"aliyun-\"}],\n"
+ + " \"suffix\": [{\"suffix\": \"-eventbridge\"}]\n" + " }\n" + "}";
+ System.out.println(st);


What is the purpose of this class?

I have addressed the code style issues and removed some code. Can you please review it again?

Alonexc · 2023-08-16T02:01:37Z

please check code style.

qqeasonchen · 2023-08-28T09:42:37Z

@xwm1992 please review it.

qqeasonchen · 2023-08-28T09:42:59Z

@pmupkin please link to the right issue.

pmupkin · 2023-09-19T15:57:04Z

please check code style.

Sure, I have fixed the code style issues.

pmupkin · 2023-09-19T16:04:44Z

@pmupkin Please link this pr to related issue

Okay, I've added some explanations and clarifications, and I've also re-linked the related issues.

codecov · 2023-10-25T07:44:33Z

Codecov Report

Merging #4365 (c8aa0e0) into master (d3f688d) will increase coverage by 0.60%.
Report is 2 commits behind head on master.
The diff coverage is 66.07%.

❗ Current head c8aa0e0 differs from pull request most recent head 56b6e39. Consider uploading reports for the commit 56b6e39 to get more accurate results

@@             Coverage Diff              @@
##             master    #4365      +/-   ##
============================================
+ Coverage     15.47%   16.07%   +0.60%     
- Complexity     1452     1532      +80     
============================================
  Files           691      710      +19     
  Lines         28106    28445     +339     
  Branches       2626     2675      +49     
============================================
+ Hits           4349     4573     +224     
- Misses        23312    23391      +79     
- Partials        445      481      +36

Files	Coverage Δ
...mesh/common/filter/condition/PrefixxCondition.java	`100.00% <100.00%> (ø)`
...sh/common/filter/condition/SpecifiedCondition.java	`100.00% <100.00%> (ø)`
...tmesh/common/filter/condition/SuffixCondition.java	`100.00% <100.00%> (ø)`
...ventmesh/common/transform/ConstantTransformer.java	`100.00% <100.00%> (ø)`
...ventmesh/common/transform/OriginalTransformer.java	`100.00% <100.00%> (ø)`
...ventmesh/common/transform/TemplateTransformer.java	`100.00% <100.00%> (ø)`
...he/eventmesh/common/transform/TransformerType.java	`100.00% <100.00%> (ø)`
...tmesh/common/filter/condition/ExistsCondition.java	`75.00% <75.00%> (ø)`
...rg/apache/eventmesh/common/transform/Variable.java	`75.00% <75.00%> (ø)`
...g/apache/eventmesh/common/filter/PatternEntry.java	`85.71% <85.71%> (ø)`
... and 9 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

xwm1992 · 2023-09-25T08:18:49Z

eventmesh-common/src/main/java/org/apache/eventmesh/common/filter/PatternEntry.java

+
+
+ public String getPatternName() {
+ return "123";


what is this return used for?

lrhkobe

LGTM

initial implementation -informal

56748c6

github-actions bot reviewed Aug 15, 2023

View reviewed changes

mxsm reviewed Aug 15, 2023

View reviewed changes

pandaapo reviewed Aug 16, 2023

View reviewed changes

qqeasonchen changed the title ~~initial implementation -informal~~ [WIP]initial implementation of filter and transform Sep 1, 2023

qqeasonchen changed the title ~~[WIP]initial implementation of filter and transform~~ [WIP][ISSUE #4033]initial implementation of filter and transform Sep 1, 2023

xwm1992 changed the title ~~[WIP][ISSUE #4033]initial implementation of filter and transform~~ [WIP][ISSUE #4043]initial implementation of filter and transform Sep 12, 2023

pmupkin added 5 commits September 19, 2023 00:21

initial implementation -informal

78ccff2

Merge branch 'apache:master' into f-t

7ebc9f7

fix code style problem

4886ba6

Merge remote-tracking branch 'origin/f-t' into f-t

0d786e5

remove unnecessary code snippets.

fae4728

pmupkin added 7 commits October 8, 2023 09:04

add license header

ff3dde5

add some dep

cebd48f

Merge branch 'master' into f-t

e110839

fix JsonPathUtils checkstyle problem

b8d9aaf

fix test class checkstyle problem

43c5ecc

Merge branch 'master' into f-t

60c7500

fix CI problem

56b6e39

xwm1992 approved these changes Oct 25, 2023

View reviewed changes

lrhkobe approved these changes Oct 25, 2023

View reviewed changes

xwm1992 merged commit a717d35 into apache:master Oct 25, 2023
7 checks passed

xwm1992 mentioned this pull request Nov 24, 2023

[Feature] Implement FilterEngine for EventMesh Filters #4577

Closed

3 tasks

xwm1992 added this to the 1.10 milestone Dec 12, 2023

pandaapo mentioned this pull request Dec 12, 2023

[ISSUE #4646] Add condition for transforming non-data values #4637

Open

pmupkin deleted the f-t branch April 29, 2024 11:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP][ISSUE #4043]initial implementation of filter and transform #4365

[WIP][ISSUE #4043]initial implementation of filter and transform #4365

pmupkin commented Aug 15, 2023 •

edited

Loading

github-actions bot left a comment

mxsm left a comment

pandaapo commented Aug 16, 2023

pandaapo Aug 16, 2023

pmupkin Sep 19, 2023

Alonexc commented Aug 16, 2023

qqeasonchen commented Aug 28, 2023

qqeasonchen commented Aug 28, 2023

pmupkin commented Sep 19, 2023 •

edited

Loading

pmupkin commented Sep 19, 2023

codecov bot commented Oct 25, 2023 •

edited

Loading

xwm1992 Sep 25, 2023

lrhkobe left a comment

[WIP][ISSUE #4043]initial implementation of filter and transform #4365

[WIP][ISSUE #4043]initial implementation of filter and transform #4365

Conversation

pmupkin commented Aug 15, 2023 • edited Loading

1. Overview

Filter

Transformer

github-actions bot left a comment

Choose a reason for hiding this comment

mxsm left a comment

Choose a reason for hiding this comment

pandaapo commented Aug 16, 2023

pandaapo Aug 16, 2023

Choose a reason for hiding this comment

pmupkin Sep 19, 2023

Choose a reason for hiding this comment

Alonexc commented Aug 16, 2023

qqeasonchen commented Aug 28, 2023

qqeasonchen commented Aug 28, 2023

pmupkin commented Sep 19, 2023 • edited Loading

pmupkin commented Sep 19, 2023

codecov bot commented Oct 25, 2023 • edited Loading

Codecov Report

xwm1992 Sep 25, 2023

Choose a reason for hiding this comment

lrhkobe left a comment

Choose a reason for hiding this comment

pmupkin commented Aug 15, 2023 •

edited

Loading

pmupkin commented Sep 19, 2023 •

edited

Loading

codecov bot commented Oct 25, 2023 •

edited

Loading