Support bi-directional service registry controllers (#171)

* [mesh]: Support bi-directional service registry controllers

* [mesh]: Add vendor

* [mesh]: Support nacos service registry

* [mesh]: Complete mesh registry syncer

* [mesh]: Use mutex pointer to copy

* [mesh]: Add more checking

* [doc]: Add NacosServiceRegistry

* [doc]: Add more document

* [mesh]: Add leader judgement for syncing

* [proxy]: Add dynamic service test

* [proxy]: Fix data race warning && Add service registry document

* Fix unit test

* Fix mod

* Fix close client in service registry drivers

* use a better name for dynamic servers

doc/controllers.md

@@ -1,8 +1,8 @@
-
 # Controllers
 
 - [Controllers](#controllers)
  - [System Controllers](#system-controllers)
+ - [ServiceRegistry](#serviceregistry)
  - [TrafficController](#trafficcontroller)
  - [RawConfigTrafficController](#rawconfigtrafficcontroller)
  - [HTTPServer](#httpserver)
@@ -17,6 +17,7 @@
  - [EtcdServiceRegistry](#etcdserviceregistry)
  - [EurekaServiceRegistry](#eurekaserviceregistry)
  - [ZookeeperServiceRegistry](#zookeeperserviceregistry)
+ - [NacosServiceRegistry](#nacosserviceregistry)
  - [Common Types](#common-types)
  - [tracing.Spec](#tracingspec)
  - [zipkin.Spec](#zipkinspec)
@@ -27,6 +28,7 @@
  - [httppipeline.Flow](#httppipelineflow)
  - [httppipeline.Filter](#httppipelinefilter)
  - [easemonitormetrics.Kafka](#easemonitormetricskafka)
+ - [nacos.ServerSpec](#nacosserverspec)
 
 As the [architecture diagram](./architecture.png) shows, the controller is the core entity to control kinds of working. There are two kinds of controllers overall:
 
@@ -44,6 +46,18 @@ The two categories are conceptual, which means they are not strict distinctions.
 
 For now, all system controllers can not be configured. It may gain this capability if necessary in the future.
 
+### ServiceRegistry
+
+We use the system controller `ServiceRegistry` as the service hub for all service registries. Current drivers are
+
+- [ConsulServiceRegistry](#consulserviceregistry)
+- [EtcdServiceRegistry](#etcdserviceregistry)
+- [EurekaServiceRegistry](#eurekaserviceregistry)
+- [ZookeeperServiceRegistry](#zookeeperserviceregistry)
+- [NacosServiceRegistry](#nacosserviceregistry)
+
+The drivers need to offer notifying change periodically, and operations to the external service registry.
+
 ### TrafficController
 
 TrafficController handles the lifecycle of HTTPServer and HTTPPipeline and their relationship. It manages the resource in a namespaced way. HTTPServer accepts incoming traffic and routes it to HTTPPipelines in the same namespace. Most other controllers could handle traffic by leverage the ability of TrafficController..
@@ -151,18 +165,18 @@ ingressClass: easegress
 httpServer:
  port: 8080
  https: false
- keepAlive: true 
- keepAliveTimeout: 60s 
+ keepAlive: true
+ keepAliveTimeout: 60s
  maxConnections: 10240
 ```
 
-| Name | Type | Description | Required |
-| ------------ | -------- | ------------------------------------------------------------------------- | --------------------- |
-| kubeConfig | string | Path of the Kubernetes configuration file. | No |
-| masterURL | string | The address of the Kubernetes API server. | No |
-| namespaces | []string | An array of Kubernetes namespaces which the IngressController needs to watch, all namespaces are watched if left empty. | No |
-| ingressClass | string | The IngressController only handles `Ingresses` with `ingressClassName` set to the value of this option. | No (default: easegress)  |
-| httpServer | [httpserver.Spec](#httpserver)  | Basic configuration for the shared HTTP traffic gate. The routing rules will be generated dynamically according to Kubernetes ingresses and should not be specified here. | Yes |
+| Name | Type  | Description  | Required  |
+| ------------ | ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------- |
+| kubeConfig | string  | Path of the Kubernetes configuration file.  | No  |
+| masterURL | string  | The address of the Kubernetes API server.  | No  |
+| namespaces | []string  | An array of Kubernetes namespaces which the IngressController needs to watch, all namespaces are watched if left empty.  | No  |
+| ingressClass | string  | The IngressController only handles `Ingresses` with `ingressClassName` set to the value of this option.  | No (default: easegress) |
+| httpServer | [httpserver.Spec](#httpserver) | Basic configuration for the shared HTTP traffic gate. The routing rules will be generated dynamically according to Kubernetes ingresses and should not be specified here. | Yes  |
 
 **Note**: IngressController uses `kubeConfig` and `masterURL` to connect to Kubernetes, at least one of them must be specified when deployed outside of a Kubernetes cluster, and both are optional when deployed inside a cluster.
 
@@ -177,14 +191,16 @@ specUpdateInterval: 10s
 heartbeatInterval: 5s
 registryType: consul
 serviceName: service-001
+externalServiceRegistry: consul-service-registry-example
 ```
 
-| Name | Type | Description | Required |
-| ----------------- | ------ | ------------------------------------------------------------------------- | --------------------- |
-| heartbeatInterval | string | Interval for one service instance reporting its heartbeat | Yes (default: 5s) |
-| registryType | string | Protocol the registry center accepts, support `eureka`, `consul`, `nacos` | Yes (default: eureka) |
-| apiPort | int | Port listening on for worker's API server | Yes (default: 13009) |
-| ingressPort | int | Port listening on for for ingress traffic | Yes (default: 13010) |
+| Name | Type | Description | Required |
+| ----------------------- | ------ | ------------------------------------------------------------------------- | --------------------- |
+| heartbeatInterval | string | Interval for one service instance reporting its heartbeat | Yes (default: 5s) |
+| registryType | string | Protocol the registry center accepts, support `eureka`, `consul`, `nacos` | Yes (default: eureka) |
+| apiPort | int | Port listening on for worker's API server | Yes (default: 13009) |
+| ingressPort | int | Port listening on for for ingress traffic | Yes (default: 13010) |
+| externalServiceRegistry | string | External service registry name | No |
 
 ### ConsulServiceRegistry
 
@@ -262,6 +278,29 @@ syncInterval: 10s
 | Prefix | string | Prefix of services | Yes (default: /) |
 | syncInterval | string | Interval to synchronize data | Yes (default: 10s) |
 
+### NacosServiceRegistry
+
+NacosServiceRegistry supports service discovery for Nacos as backend. The config looks like:
+
+```yaml
+kind: NacosServiceRegistry
+name: nacos-service-registry-example
+syncInterval: 10s
+servers:
+ - scheme: http
+ port: 8848
+ contextPath: /nacos
+ ipAddr: 127.0.0.1
+```
+
+| Name | Type | Description | Required |
+| ------------ | ------------------------------------- | ---------------------------- | ------------------ |
+| servers | [][nacosServerSpec](#nacosserverspec) | Servers of Nacos | Yes |
+| syncInterval | string | Interval to synchronize data | Yes (default: 10s) |
+| namespace | string | The namespace of Nacos | No |
+| username | string | The username of client | No |
+| password | string | The password of client | No |
+
 ## Common Types
 
 ### tracing.Spec
@@ -324,9 +363,9 @@ There must be at least one of `values` and `regexp`.
 
 ### httppipeline.Flow
 
-| Name | Type | Description  | Required |
-| ------ | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- |
-| filter | string | The filter name  | Yes |
+| Name | Type | Description | Required |
+| ------ | ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- |
+| filter | string | The filter name | Yes |
 | jumpIf | map[string]string | Jump to another filter conditionally, the key is the result of the current filter, the value is the jumping filter name. `END` is the built-in value for the ending of the pipeline | No |
 
 ### httppipeline.Filter
@@ -345,3 +384,12 @@ The self-defining specification of each filter references to [filters](./filters
 | ------- | -------- | ---------------- | ----------------------------- |
 | brokers | []string | Broker addresses | Yes (default: localhost:9092) |
 | topic | string | Produce topic | Yes |
+
+### nacos.ServerSpec
+
+| Name | Type | Description | Required |
+| ----------- | ------ | -------------------------------------------- | -------- |
+| ipAddr | string | The ip address | Yes |
+| port | uint16 | The port | Yes |
+| scheme | string | The scheme of protocol (support http, https) | No |
+| contextPath | string | The context path | No |

example/config/etcd-service-registry-example.yaml

@@ -1,5 +1,5 @@
 kind: EtcdServiceRegistry
 name: etcd-service-registry-example
-endpoints: ['127.0.0.1:12379']
+endpoints: ['127.0.0.1:2379']
 prefix: "/services/"
-cacheTimeout: 10s
+cacheTimeout: 5s

example/config/http-pipeline-example.yaml

@@ -101,6 +101,8 @@ filters:
  X-Filter:
  exact: candidate
  mainPool:
+ serviceRegistry: nacos-service-registry-example
+ serviceName: service-001
  serversTags: ["v2"]
  servers:
  - url: https://127.0.0.1:9095

example/config/mesh-controller-example.yaml

@@ -1,6 +1,5 @@
 name: mesh-controller-example
 kind: MeshController
-specUpdateInterval: 10s
 heartbeatInterval: 5s
 registryType: consul
-serviceName: service-001
+externalServiceRegistry: nacos-service-registry-example

example/config/nacos-service-registry-example.yaml

@@ -0,0 +1,8 @@
+kind: NacosServiceRegistry
+name: nacos-service-registry-example
+syncInterval: 10s
+servers:
+ - scheme: http
+ port: 8848
+ contextPath: /nacos
+ ipAddr: 127.0.0.1

example/config/server-001-instance-001.yaml

@@ -0,0 +1,7 @@
+registryName: etcd-service-registry-example
+serviceName: service-001
+instanceID: intance-001
+scheme: http
+hostIP: 127.0.0.1
+port: 9091
+tags: [v1]

example/config/server-001-instance-002.yaml

@@ -0,0 +1,7 @@
+registryName: etcd-service-registry-example
+serviceName: service-001
+instanceID: intance-002
+scheme:
+hostIP: 127.0.0.1
+port: 9092
+tags: [v2]

example/config/server-001-instance-003.yaml

@@ -0,0 +1,7 @@
+registryName: etcd-service-registry-example
+serviceName: service-001
+instanceID: intance-003
+scheme: http
+hostIP: 127.0.0.1
+port: 9093
+tags: [v3]

example/config/server-002-instance-001.yaml

@@ -0,0 +1,5 @@
+registryName: mesh-controller-example
+serviceName: service-002
+instanceID: instance-001
+ip: 127.0.0.1
+port: 9094

go.mod

@@ -16,6 +16,7 @@ require (
  github.com/go-chi/chi/v5 v5.0.3
  github.com/go-zookeeper/zk v1.0.2
  github.com/golang-jwt/jwt v3.2.1+incompatible
+ github.com/google/uuid v1.2.0
  github.com/gorilla/websocket v1.4.2
  github.com/hashicorp/consul/api v1.8.1
  github.com/hashicorp/golang-lru v0.5.4

go.sum

@@ -147,6 +147,7 @@ github.com/alecthomas/units v0.0.0-20190717042225-c3de453c63f4/go.mod h1:ybxpYRF
 github.com/alecthomas/units v0.0.0-20190924025748-f65c72e2690d h1:UQZhZ2O0vMHr2cI+DC1Mbh0TJxzA3RcLoMsFw+aXw7E=
 github.com/alecthomas/units v0.0.0-20190924025748-f65c72e2690d/go.mod h1:rBZYJk541a8SKzHPHnH3zbiI+7dagKZ0cgpgrD7Fyho=
 github.com/alexflint/go-filemutex v0.0.0-20171022225611-72bdc8eae2ae/go.mod h1:CgnQgUtFrFz9mxFNtED3jI5tLDjKlOM+oUF/sTk6ps0=
+github.com/aliyun/alibaba-cloud-sdk-go v1.61.18 h1:zOVTBdCKFd9JbCKz9/nt+FovbjPFmb7mUnp8nH9fQBA=
 github.com/aliyun/alibaba-cloud-sdk-go v1.61.18/go.mod h1:v8ESoHo4SyHmuB4b1tJqDHxfTGEciD+yhvOU/5s1Rfk=
 github.com/andreyvit/diff v0.0.0-20170406064948-c7f18ee00883/go.mod h1:rCTlJbsFo29Kk6CurOXKm700vrz8f0KW0JNfpkRJY/8=
 github.com/anmitsu/go-shlex v0.0.0-20161002113705-648efa622239/go.mod h1:2FmKhYUyUczH0OGQWaF5ceTx0UBShxjsH6f8oGKYe2c=
@@ -196,6 +197,7 @@ github.com/bmizerany/perks v0.0.0-20141205001514-d9a9656a3a4b/go.mod h1:ac9efd0D
 github.com/bradfitz/go-smtpd v0.0.0-20170404230938-deb6d6237625/go.mod h1:HYsPBTaaSFSlLx/70C2HPIMNZpVV8+vt/A+FMnYP11g=
 github.com/bshuster-repo/logrus-logstash-hook v0.4.1/go.mod h1:zsTqEiSzDgAa/8GZR7E1qaXrhYNDKBYy5/dWPTIflbk=
 github.com/buger/jsonparser v0.0.0-20180808090653-f4dd9f5a6b44/go.mod h1:bbYlZJ7hK1yFx9hf58LP0zeX7UjIGs20ufpu3evjr+s=
+github.com/buger/jsonparser v0.0.0-20181115193947-bf1c66bbce23 h1:D21IyuvjDCshj1/qq+pCNd3VZOAEI9jy6Bi131YlXgI=
 github.com/buger/jsonparser v0.0.0-20181115193947-bf1c66bbce23/go.mod h1:bbYlZJ7hK1yFx9hf58LP0zeX7UjIGs20ufpu3evjr+s=
 github.com/bugsnag/bugsnag-go v0.0.0-20141110184014-b1d153021fcd/go.mod h1:2oa8nejYd4cQ/b0hMIopN0lCRxU0bueqREvZLWFrtK8=
 github.com/bugsnag/osext v0.0.0-20130617224835-0dd3f918b21b/go.mod h1:obH5gd0BsqsP2LwDJ9aOkm/6J86V6lyAXCoQWGw3K50=
@@ -431,6 +433,7 @@ github.com/facebookgo/stack v0.0.0-20160209184415-751773369052 h1:JWuenKqqX8nojt
 github.com/facebookgo/stack v0.0.0-20160209184415-751773369052/go.mod h1:UbMTZqLaRiH3MsBH8va0n7s1pQYcu3uTb8G4tygF4Zg=
 github.com/facebookgo/subset v0.0.0-20200203212716-c811ad88dec4 h1:7HZCaLC5+BZpmbhCOZJ293Lz68O7PYrF2EzeiFMwCLk=
 github.com/facebookgo/subset v0.0.0-20200203212716-c811ad88dec4/go.mod h1:5tD+neXqOorC30/tWg0LCSkrqj/AR6gu8yY8/fpw1q0=
+github.com/fastly/go-utils v0.0.0-20180712184237-d95a45783239 h1:Ghm4eQYC0nEPnSJdVkTrXpu9KtoVCSo1hg7mtI7G9KU=
 github.com/fastly/go-utils v0.0.0-20180712184237-d95a45783239/go.mod h1:Gdwt2ce0yfBxPvZrHkprdPPTTS3N5rwmLE8T22KBXlw=
 github.com/fatih/color v1.7.0/go.mod h1:Zm6kSWBoL9eyXnKyktHP6abPY2pDugNf5KwzbycvMj4=
 github.com/fatih/color v1.9.0/go.mod h1:eQcE1qtQxscV5RaZvpXrrb8Drkc3/DdQ+uUYCNjL+zU=
@@ -462,6 +465,7 @@ github.com/globalsign/mgo v0.0.0-20180905125535-1ca0a4f7cbcb/go.mod h1:xkRDCp4j0
 github.com/globalsign/mgo v0.0.0-20181015135952-eeefdecb41b8/go.mod h1:xkRDCp4j0OGD1HRkm4kmhM+pmpv3AKq5SU7GMg4oO/Q=
 github.com/go-chi/chi/v5 v5.0.3 h1:khYQBdPivkYG1s1TAzDQG1f6eX4kD2TItYVZexL5rS4=
 github.com/go-chi/chi/v5 v5.0.3/go.mod h1:DslCQbL2OYiznFReuXYUmQ2hGd1aDpCnlMNITLSKoi8=
+github.com/go-errors/errors v1.0.1 h1:LUHzmkK3GUKUrL/1gfBUxAHzcev3apQlezX/+O7ma6w=
 github.com/go-errors/errors v1.0.1/go.mod h1:f4zRHt4oKfwPJE5k8C9vpYG+aDHdBFUsgrm6/TyX73Q=
 github.com/go-gl/glfw v0.0.0-20190409004039-e6da0acd62b1/go.mod h1:vR7hzQXu2zJy9AVAgeJqvqgH9Q5CA+iKCZ2gyEVpxRU=
 github.com/go-gl/glfw/v3.3/glfw v0.0.0-20191125211704-12ad95a8df72/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8=
@@ -785,6 +789,7 @@ github.com/jcmturner/gokrb5/v8 v8.4.2 h1:6ZIM6b/JJN0X8UM43ZOM6Z4SJzla+a/u7scXFJz
 github.com/jcmturner/gokrb5/v8 v8.4.2/go.mod h1:sb+Xq/fTY5yktf/VxLsE3wlfPqQjp0aWNYyvBVK62bc=
 github.com/jcmturner/rpc/v2 v2.0.3 h1:7FXXj8Ti1IaVFpSAziCZWNzbNuZmnvw/i6CqLNdWfZY=
 github.com/jcmturner/rpc/v2 v2.0.3/go.mod h1:VUJYCIDm3PVOEHw8sgt091/20OJjskO/YJki3ELg/Hc=
+github.com/jehiah/go-strftime v0.0.0-20171201141054-1d33003b3869 h1:IPJ3dvxmJ4uczJe5YQdrYB16oTJlGSC/OyZDqUk9xX4=
 github.com/jehiah/go-strftime v0.0.0-20171201141054-1d33003b3869/go.mod h1:cJ6Cj7dQo+O6GJNiMx+Pa94qKj+TG8ONdKHgMNIyyag=
 github.com/jellevandenhooff/dkim v0.0.0-20150330215556-f50fe3d243e1/go.mod h1:E0B/fFc00Y+Rasa88328GlI/XbtyysCtTHZS8h7IrBU=
 github.com/jessevdk/go-flags v1.4.0/go.mod h1:4FA24M0QyGHXBuZZK/XkWh8h0e1EYbRYJSGM75WSRxI=
@@ -846,8 +851,11 @@ github.com/kr/pty v1.1.5/go.mod h1:9r2w37qlBe7rQ6e1fg1S/9xpWHSnaqNdHD3WcMdbPDA=
 github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
 github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
 github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
+github.com/lestrrat/go-envload v0.0.0-20180220120943-6ed08b54a570 h1:0iQektZGS248WXmGIYOwRXSQhD4qn3icjMpuxwO7qlo=
 github.com/lestrrat/go-envload v0.0.0-20180220120943-6ed08b54a570/go.mod h1:BLt8L9ld7wVsvEWQbuLrUZnCMnUmLZ+CGDzKtclrTlE=
+github.com/lestrrat/go-file-rotatelogs v0.0.0-20180223000712-d3151e2a480f h1:sgUSP4zdTUZYZgAGGtN5Lxk92rK+JUFOwf+FT99EEI4=
 github.com/lestrrat/go-file-rotatelogs v0.0.0-20180223000712-d3151e2a480f/go.mod h1:UGmTpUd3rjbtfIpwAPrcfmGf/Z1HS95TATB+m57TPB8=
+github.com/lestrrat/go-strftime v0.0.0-20180220042222-ba3bf9c1d042 h1:Bvq8AziQ5jFF4BHGAEDSqwPW1NJS3XshxbRCxtjFAZc=
 github.com/lestrrat/go-strftime v0.0.0-20180220042222-ba3bf9c1d042/go.mod h1:TPpsiPUEh0zFL1Snz4crhMlBe60PYxRHr5oFF3rRYg0=
 github.com/liggitt/tabwriter v0.0.0-20181228230101-89fcab3d43de h1:9TO3cAIGXtEhnIaL+V+BEER86oLrvS+kWobKpbJuye0=
 github.com/liggitt/tabwriter v0.0.0-20181228230101-89fcab3d43de/go.mod h1:zAbeS9B/r2mtpb6U+EI2rYA5OAXxsYw6wTamcNW+zcE=
@@ -1256,6 +1264,7 @@ github.com/tarm/serial v0.0.0-20180830185346-98f6abe2eb07/go.mod h1:kDXzergiv9cb
 github.com/tchap/go-patricia v2.2.6+incompatible/go.mod h1:bmLyhP68RS6kStMGxByiQ23RP/odRBOTVjwp2cDyi6I=
 github.com/tcnksm/go-httpstat v0.2.1-0.20191008022543-e866bb274419 h1:elOIj31UL4RZWgLfLV4pWZA0j5QqGO95/Dll2WIwOZU=
 github.com/tcnksm/go-httpstat v0.2.1-0.20191008022543-e866bb274419/go.mod h1:s3JVJFtQxtBEBC9dwcdTTXS9xFnM3SXAZwPG41aurT8=
+github.com/tebeka/strftime v0.1.3 h1:5HQXOqWKYRFfNyBMNVc9z5+QzuBtIXy03psIhtdJYto=
 github.com/tebeka/strftime v0.1.3/go.mod h1:7wJm3dZlpr4l/oVK0t1HYIc4rMzQ2XJlOMIUJUJH6XQ=
 github.com/tidwall/gjson v1.8.0 h1:Qt+orfosKn0rbNTZqHYDqBrmm3UDA4KRkv70fDzG+PQ=
 github.com/tidwall/gjson v1.8.0/go.mod h1:5/xDoumyyDNerp2U36lyolv46b3uF/9Bu6OfyQ9GImk=
@@ -1270,6 +1279,7 @@ github.com/tmc/grpc-websocket-proxy v0.0.0-20201229170055-e5319fda7802 h1:uruHq4
 github.com/tmc/grpc-websocket-proxy v0.0.0-20201229170055-e5319fda7802/go.mod h1:ncp9v5uamzpCO7NfCPTXjqaC+bZgJeR0sMTm6dMHP7U=
 github.com/tomasen/realip v0.0.0-20180522021738-f0c99a92ddce h1:fb190+cK2Xz/dvi9Hv8eCYJYvIGUTN2/KLq1pT6CjEc=
 github.com/tomasen/realip v0.0.0-20180522021738-f0c99a92ddce/go.mod h1:o8v6yHRoik09Xen7gje4m9ERNah1d1PPsVq1VEx9vE4=
+github.com/toolkits/concurrent v0.0.0-20150624120057-a4371d70e3e3 h1:kF/7m/ZU+0D4Jj5eZ41Zm3IH/J8OElK1Qtd7tVKAwLk=
 github.com/toolkits/concurrent v0.0.0-20150624120057-a4371d70e3e3/go.mod h1:QDlpd3qS71vYtakd2hmdpqhJ9nwv6mD6A30bQ1BPBFE=
 github.com/tsenart/go-tsz v0.0.0-20180814232043-cdeb9e1e981e/go.mod h1:SWZznP1z5Ki7hDT2ioqiFKEse8K9tU2OUvaRI0NeGQo=
 github.com/tsenart/go-tsz v0.0.0-20180814235614-0bd30b3df1c3/go.mod h1:SWZznP1z5Ki7hDT2ioqiFKEse8K9tU2OUvaRI0NeGQo=