Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add pipeline route for mqttproxy #453

Merged
merged 15 commits into from
Jan 12, 2022
Merged

add pipeline route for mqttproxy #453

merged 15 commits into from
Jan 12, 2022

Conversation

suchen-sci
Copy link
Contributor

@suchen-sci suchen-sci commented Jan 5, 2022

This pr add Pipeline route for MQTTProxy and also make backend Kafka a MQTT filter.

New version yaml:
Now MQTTProxy can route MQTT packets to different pipelines based on the packet type. Currently support: Connect, Disconnect, Publish, Subscribe, Unsubscribe.

kind: MQTTProxy
name: mqttproxy
port: 1883
auth:
  - userName: test
    passBase64: dGVzdA==
  - userName: admin
    passBase64: YWRtaW4=
pipelines:
  - name: publish-pipeline
    packetType: Publish
  - name: connect-pipeline
    packetType: Connect
name: publish-pipeline
kind: Pipeline 
protocol: MQTT
flow:
  - filter: topicMapper
  - filter: kafka 
filters:
  - name: topicMapper
    kind: TopicMapper
    ...
  - name: kafka
    kind: Kafka
    backend: [":9093"]

@codecov-commenter
Copy link

codecov-commenter commented Jan 5, 2022

Codecov Report

Merging #453 (5e38e91) into main (42801f2) will decrease coverage by 0.05%.
The diff coverage is 82.13%.

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #453      +/-   ##
==========================================
- Coverage   80.57%   80.52%   -0.06%     
==========================================
  Files          70       74       +4     
  Lines        8171     8353     +182     
==========================================
+ Hits         6584     6726     +142     
- Misses       1230     1261      +31     
- Partials      357      366       +9     
Impacted Files Coverage Δ
pkg/object/mqttproxy/spec.go 100.00% <ø> (ø)
pkg/object/mqttproxy/topic.go 100.00% <ø> (ø)
pkg/object/pipeline/mock.go 80.64% <0.00%> (-6.32%) ⬇️
pkg/filter/topicmapper/topicmapper.go 48.64% <48.64%> (ø)
pkg/filter/kafka/kafka.go 62.90% <62.90%> (ø)
pkg/object/mqttproxy/broker.go 74.33% <78.37%> (-0.72%) ⬇️
pkg/filter/mqttclientauth/mqttauth.go 83.56% <83.56%> (ø)
pkg/object/mqttproxy/client.go 80.54% <85.71%> (+3.37%) ⬆️
pkg/filter/connectcontrol/connectcontrol.go 96.10% <100.00%> (ø)
pkg/filter/kafka/topic.go 100.00% <100.00%> (ø)
... and 7 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 42801f2...5e38e91. Read the comment docs.

@suchen-sci suchen-sci marked this pull request as draft January 5, 2022 09:32
@suchen-sci suchen-sci marked this pull request as ready for review January 6, 2022 03:29
@zhao-kun
Copy link
Collaborator

zhao-kun commented Jan 6, 2022

Suggest change YAML formation to:

rules:
  - when:
      packetType: Publish
    pipleline: publish-pipeline

For now, we just support the packetType as the only condition to dispatch the data to the pipeline. In the future, we could add extra conditions in when, such as topic start with /root/*, etc...

substitute keyword pipelines to rules just in order to align the YAML contents to HTTPPipeline.

@suchen-sci
Copy link
Contributor Author

updated!

pkg/context/mqttcontext.go Outdated Show resolved Hide resolved
pkg/filter/authentication/auth.go Outdated Show resolved Hide resolved
pkg/filter/authentication/auth.go Outdated Show resolved Hide resolved
pkg/object/pipeline/mock.go Show resolved Hide resolved
Copy link
Collaborator

@zhao-kun zhao-kun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall, look good to me, except for decoupling the Kafka plugin

pkg/filter/kafka/kafka_test.go Outdated Show resolved Hide resolved
pkg/filter/kafka/kafka.go Outdated Show resolved Hide resolved
pkg/filter/kafka/spec.go Outdated Show resolved Hide resolved
@zhao-kun
Copy link
Collaborator

zhao-kun commented Jan 12, 2022

I have some jobs that were suspended for the PR. @localvar please help review the PR and push forward the progress of the merging when you are free

pkg/context/mqttcontext.go Outdated Show resolved Hide resolved
pkg/context/mqttpublish.go Outdated Show resolved Hide resolved
pkg/filter/kafka/kafka.go Outdated Show resolved Hide resolved
pkg/filter/mqttclientauth/mqttauth.go Outdated Show resolved Hide resolved
pkg/filter/topicmapper/topic.go Outdated Show resolved Hide resolved
Copy link
Contributor

@samutamm samutamm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@zhao-kun zhao-kun merged commit 7548804 into easegress-io:main Jan 12, 2022
@suchen-sci suchen-sci deleted the update-mqttproxy-pipeline-schema branch January 26, 2022 07:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants