Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ISSUE #4787]integration connector #4790

Closed
wants to merge 3 commits into from
Closed

Conversation

Ish1yu
Copy link
Contributor

@Ish1yu Ish1yu commented Mar 15, 2024

Fixes #4787

########
UnMergeable!!! its demo ,不能合并这只是用于讨论
########

Motivation

Integrate connector with minimal modification

Very simple integration using connecter

Modifications

Integrating 'connecter' means that 'connecter' has to have another running mode, and integrating into 'runtime' has some problems to solve
集成 connecter ,意味着 connecter 要有另一种运行模式,集成到 runtime 中有一些问题需要解决


The first problem that needs to be solved is that the main method is not called, but connecter is a plug-in module, and it is troublesome to hard-code main once for each plug-in. So the method I think of is that all the connecter module's startup class inherits a EmbeddableConnector interface, and uses the Eventmesh-SPI to load, so that the changes are brought
EmbeddableConnector
All startup classes in Connector (in this case, FileConnectServer)
All files referred to by SPI in Connector
首先需要解决的问题就是 main 方法没有被调用,但是 connecter 是插件化的模块,为每一个插件都硬编码一次 main 这样很麻烦,所以我想到的方法是所有 connecter 模块的启动类都继承一个EmbeddableConnector 接口,使用 EventmeshSPI 进行加载,这样带来的修改就有
EmbeddableConnector
所有 Connector 中的启动类(这个例子中的启动类是 FileConnectServer
所有 ConnectorSPI 涉及的文件


Second, the original interaction between connector and runtime uses tcp protocol, and the purpose of integration is to reduce serialization and communication consumption. Therefore, interaction needs to be realized in other ways. The method I think is to use the form of service. connector itself only has two roles, sink and source, which are publish and subscribe in terms of function, so this service needs to use publish and subscribe functions. Meanwhile, in order to preserve the original structure and design to the greatest extent, the connector uses a tcp adapter (the adapter does not implement network communication in the middle). Instead, the service is called.) So the design changes are
IInnerPubSubService Indicates the interface of the service
InnerPubSubService implementation
IntegrationCloudEventTCPClientAdapter TCP adapter
Application must have a static variable holding the service
The SinkWorker passes the service to the tcp adapter
SourceWorker transfers the service tcp adapter
其次需要解决的就是原有 connectorruntime 之间的交互使用的是 tcp协议 而集成的目的就是减少序列化和通信消耗,所以需要通过其他方式对交互进行实现,我想到的方法是使用服务的形式, connector 本身其实只有两种角色,sink source ,从功能来说就是发布和订阅,那么这个服务就需要用发布和订阅的功能即可,同时为了最大程度保留原有结构和设计,使用tcp适配器(适配器中间不在实现网络通信,而是对服务进行调用)所以设计的修改有
IInnerPubSubService 服务的接口
InnerPubSubService 服务的实现
IntegrationCloudEventTCPClientAdapter tcp适配器
Application 要有静态变量持有服务
SinkWorker 传递服务给tcp适配器
SourceWorker 传递服务tcp适配器


Unfinished changes:
The implementation of InnerPubSubService
未完成的修改:
InnerPubSubService 的具体实现


达成的效果:
只要 gradle 引用了对应的 connector 插件就会自动处理 内嵌 connector 相关逻辑
Results achieved:
The built-in connector logic is automatically handled whenever gradle references the corresponding connector plug-in

Copy link
Member

@pandaapo pandaapo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since the demo code is just a general idea, I've only looked at the general idea, so I'll leave some minor review comments for now. I think it should be possible to realize "when start runtime, if there are local Connectors, they will be started together with SPI mechanism, and the communication method will be intra-process communication". I agree with this idea, and look forward to hearing from more people in the community.
But there is a question, should we consider this case: the local Connector has already started by itself?

因为该demo代码只是大致的思路,所以只看了个大概,一些细小的review意见就暂时不提了。我觉得该思路应该可以实现“启动runtime时,如果本地有Connectors会利用SPI机制一并启动,并且通信方式将是进程内通信”。我赞同该思路,同时期待社区更多人的看法。
但是有个疑问,要不要考虑这种情况:本地的Connector已经自己启动过了?

@Ish1yu
Copy link
Contributor Author

Ish1yu commented Mar 18, 2024

Sorry, I have a bad cold these days, so I think it may be necessary to use a form like file lock. After connector is activated, the Worker will generate a file named Connector-instance-mian at a fixed location. When the connector is booted either integrally or separately, it will get an error that cannot be started if the file exists (init timing), and delete the file when the connector stops (shutdown timing). The only drawback is the need to manually delete this file if the connector stops unexpectedly (such as when the process is killed).
不好意思,这几天重感冒,那我想可能需要使用比如类似文件锁的形式,当connector启动后Worker在某一固定位置生成一个名为connector-instance-mian类名的文件,当无论以集成的方式还是单独启动都会去获取这个文件如果文件存在那么就报错不可以启动(init时机),当connector停止时删除这个文件(shutdown时机),唯一的缺陷是当connector意外停止时需要手动删除这个文件(比如进程被杀了)

@pandaapo
Copy link
Member

Sorry, I have a bad cold these days, so I think it may be necessary to use a form like file lock. After connector is activated, the Worker will generate a file named Connector-instance-mian at a fixed location. When the connector is booted either integrally or separately, it will get an error that cannot be started if the file exists (init timing), and delete the file when the connector stops (shutdown timing). The only drawback is the need to manually delete this file if the connector stops unexpectedly (such as when the process is killed).
不好意思,这几天重感冒,那我想可能需要使用比如类似文件锁的形式,当connector启动后Worker在某一固定位置生成一个名为connector-instance-mian类名的文件,当无论以集成的方式还是单独启动都会去获取这个文件如果文件存在那么就报错不可以启动(init时机),当connector停止时删除这个文件(shutdown时机),唯一的缺陷是当connector意外停止时需要手动删除这个文件(比如进程被杀了)

I see in the issue that you have raised a lot of queries about the community's planning for this part of the demand, so let's leave this minor detail aside for now. I'll keep an eye on these queries as well #4787 (comment) .

我看到issue中您提了很多关于社区对这部分需求的规划的疑问,所以这个次要的细节问题暂不考虑吧。对于这些疑问(#4787 (comment) ),我也会继续关注。

@Ish1yu Ish1yu closed this Apr 7, 2024
@Pil0tXia
Copy link
Member

Pil0tXia commented Apr 7, 2024

@Ish1yu

Now we have a detailed issue #4816 describing the connector and function integration design, which may greatly help your subsequent work of this PR. So any communication with community is welcomed~

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Feature] Embed the connector into EventMesh Server for DTS
3 participants