add WebSocketProxy and remove WebSocketServer (#802)

* update mux to support hijack * add websocketproxy * basic feature is completed * refactor to reduce code duplication * go mod tidy * fix failed test cases * improve test coverage * update documentation * remove websocket server * go mod tidy * add TODO for later refactor
easegress-io · Sep 28, 2022 · 5582774 · 5582774
1 parent 65adc61
commit 5582774
Show file tree

Hide file tree

Showing 23 changed files with 1,078 additions and 1,179 deletions.
diff --git a/doc/cookbook/websocket.md b/doc/cookbook/websocket.md
@@ -8,74 +8,81 @@
 
 ## Background
 
-- Reverse proxy is one of most popular features of Easegress and it's suitable for many production and development scenarios.
+- Reverse proxy is one of most popular features of Easegress and it's suitable
+ for many production and development scenarios.
 
-- In reversed proxy use cases, `WebSocket` is a widely used protocol for a full-duplex communication solution between client and server. WebSocket relies on TCP.[1]
+- In reversed proxy use cases, `WebSocket` is a widely used protocol for a
+ full-duplex communication solution between client and server. WebSocket
+ relies on TCP.[1]
 
 - Many reverse proxy support `WebSocket`,e.g., NGINX[2], Traefik, and so on.
 
 ## Design
 
-- WebSocket server of Easegress is called `WebSocketServer` and it belongs to the group of BusinessControllers.
-
-- Easegress uses `github.com/gorilla/websocket` to implement `WebSocket` client since it has rich features supported and a quite active community (15k star/2.5k fork). Gorilla supports some useful features (`receive fragmented message` and `send close message`) that do not exist in Go's standard WebSocket library `golang.org/x/net/websocket` [3], which make it a natural choice for Easegress WebSocketServer.
+- The `WebSocketProxy` is a filter of Easegress, and can be put into a pipeline.
+- Easegress uses `github.com/golang/x/net/websocket` to implement
+ `WebSocketProxy` filter.
 
 1. Spec
 
- Example1: `http` and `ws`
-
- ```yaml
- kind: WebSocketServer
- name: websocketSvr
- https: false # client need to use http/https firstly for connection upgrade 
- certBase64:
- keyBase64:
- port: 10020 # proxy servers listening port
-
- backend: ws:https://localhost:3001 # the reserved proxy target
- # Easegress will exame the backend URL's scheme, If it starts with `wss`,
- # then `wssCerBase64` and `wssKeyBase64` must not be empty
-
- wssCertBase64: # wss backend certificate in base64 format
- wssKeyBase64: # wss backend key in base64 format
- ```
-
- Example2: `https` and `wss`
-
- ```yaml
- kind: WebSocketServer
- name: websocketSvr
- https: true 
- certBase64: your-cert-base64
- keyBase64: your-key-base64
- port: 10020
- backend: wss:https://localhost:3001
- wssCertBase64: your-cert-wss-base64
- wssKeyBase64: your-key-wss-base64
- ```
+* Pipeline with a `WebSocketProxy` filter:
+
+```yaml
+ name: websocket-pipeline
+ kind: Pipeline
+
+ flow:
+ - filter: wsproxy
+
+ filters:
+ - kind: WebSocketProxy
+ name: wsproxy
+ defaultOrigin: http:https://127.0.0.1/hello
+ pools:
+ - servers:
+ - url: ws:https://127.0.0.1:12345
+ ```
+
+* HTTPServer to route traffic to the websocket-pipeline:
+
+```yaml
+name: demo-server
+kind: HTTPServer
+port: 8080
+rules:
+- paths:
+ path: /ws
+ clientMaxBodySize: -1 # REQUIRED!
+ backend: websocket-pipeline
+```
+
+Note: `clientMaxBodySize` must be `-1`.
 
 2. Request sequence
 
  ```none
  +--------------+ +--------------+ +--------------+ 
  | | 1 | | 2 | | 
  | client +--------------->| Easegress +--------------->| websocket |
- | |<---------------+(WebSocketSvr)|<---------------+ backend | 
+ | |<---------------+ |<---------------+ backend | 
  | | 4 | | 3 | |
  +--------------+ +--------------+ +--------------+
  ```
 
 3. Headers
 
- We copy all headers from your HTTP request to websocket backend, except ones used by `gorilla` package to build connection. Based on [4], we also add `X-Forwarded-For`, `X-Forwarded-Host`, `X-Forwarded-Proto` to http headers that send to websocket backend.
+We copy all headers from your HTTP request to websocket backend, except
+ones used by websocket package to build connection. Based on [3], we also
+add `X-Forwarded-For`, `X-Forwarded-Host`, `X-Forwarded-Proto` to http
+headers that send to websocket backend.
 
-  > note: `gorilla` use `Upgrade`, `Connection`, `Sec-Websocket-Key`, `Sec-Websocket-Version`, `Sec-Websocket-Extensions` and `Sec-Websocket-Protocol` in http headers to set connection.
+> note: websocket use `Upgrade`, `Connection`, `Sec-Websocket-Key`,
+ `Sec-Websocket-Version`, `Sec-Websocket-Extensions` and
+ `Sec-Websocket-Protocol` in http headers to set connection.
 
 ## Example
 
-1. Create a WebSocket proxy for Easegress: `egctl object create -f websocket.yaml`. Here we use `Example1` as example, which will transfer requests from `easegress-ip:10020` to `ws:https://localhost:3001`.
-
-2. Send request
+1. Send request
 
  ```bash
  curl --include \
@@ -85,14 +92,14 @@
  --header "Host: 127.0.0.1:10020" \
  --header "Sec-WebSocket-Key: your-key-here" \
  --header "Sec-WebSocket-Version: 13" \
- http:https://127.0.0.1:10081/
+ http:https://127.0.0.1:8080/
  ```
 
-3. This request to `WebSocketServer` `easegress-ip:10081` will be transferred to websocket backend `ws:https://localhost:3001`.
+2. This request to Easegress will be forwarded to websocket backend
+ `ws:https://127.0.0.1:12345`.
 
 ## References
 
 1. <https://datatracker.ietf.org/doc/html/rfc6455>
 2. <https://www.nginx.com/blog/websocket-nginx/>
-3. <https://github.com/gorilla/websocket>
-4. <https://docs.oracle.com/en-us/iaas/Content/Balance/Reference/httpheaders.htm>
+3. <https://docs.oracle.com/en-us/iaas/Content/Balance/Reference/httpheaders.htm>
diff --git a/doc/reference/filters.md b/doc/reference/filters.md
@@ -4,54 +4,57 @@
  - [Proxy](#proxy)
  - [Configuration](#configuration)
  - [Results](#results)
- - [CORSAdaptor](#corsadaptor)
+ - [WebSocketProxy](#websocketproxy)
  - [Configuration](#configuration-1)
  - [Results](#results-1)
- - [Fallback](#fallback)
+ - [CORSAdaptor](#corsadaptor)
  - [Configuration](#configuration-2)
  - [Results](#results-2)
- - [Mock](#mock)
+ - [Fallback](#fallback)
  - [Configuration](#configuration-3)
  - [Results](#results-3)
- - [RemoteFilter](#remotefilter)
+ - [Mock](#mock)
  - [Configuration](#configuration-4)
  - [Results](#results-4)
- - [RequestAdaptor](#requestadaptor)
+ - [RemoteFilter](#remotefilter)
  - [Configuration](#configuration-5)
  - [Results](#results-5)
- - [RequestBuilder](#requestbuilder)
+ - [RequestAdaptor](#requestadaptor)
  - [Configuration](#configuration-6)
  - [Results](#results-6)
- - [RateLimiter](#ratelimiter)
+ - [RequestBuilder](#requestbuilder)
  - [Configuration](#configuration-7)
  - [Results](#results-7)
- - [ResponseAdaptor](#responseadaptor)
+ - [RateLimiter](#ratelimiter)
  - [Configuration](#configuration-8)
  - [Results](#results-8)
- - [ResponseBuilder](#responsebuilder)
+ - [ResponseAdaptor](#responseadaptor)
  - [Configuration](#configuration-9)
  - [Results](#results-9)
- - [Validator](#validator)
+ - [ResponseBuilder](#responsebuilder)
  - [Configuration](#configuration-10)
  - [Results](#results-10)
- - [WasmHost](#wasmhost)
+ - [Validator](#validator)
  - [Configuration](#configuration-11)
  - [Results](#results-11)
- - [Kafka](#kafka)
+ - [WasmHost](#wasmhost)
  - [Configuration](#configuration-12)
  - [Results](#results-12)
- - [HeaderToJSON](#headertojson)
+ - [Kafka](#kafka)
  - [Configuration](#configuration-13)
  - [Results](#results-13)
- - [CertExtractor](#certextractor)
+ - [HeaderToJSON](#headertojson)
  - [Configuration](#configuration-14)
  - [Results](#results-14)
- - [HeaderLookup](#headerlookup)
+ - [CertExtractor](#certextractor)
  - [Configuration](#configuration-15)
  - [Results](#results-15)
- - [ResultBuilder](#resultbuilder)
+ - [HeaderLookup](#headerlookup)
  - [Configuration](#configuration-16)
  - [Results](#results-16)
+ - [ResultBuilder](#resultbuilder)
+ - [Configuration](#configuration-17)
+ - [Results](#results-17)
  - [Common Types](#common-types)
  - [pathadaptor.Spec](#pathadaptorspec)
  - [pathadaptor.RegexpReplace](#pathadaptorregexpreplace)
@@ -66,6 +69,7 @@
  - [urlrule.URLRule](#urlruleurlrule)
  - [proxy.Compression](#proxycompression)
  - [proxy.MTLS](#proxymtls)
+ - [websocketproxy.WebSocketServerPoolSpec](#websocketproxywebsocketserverpoolspec)
  - [mock.Rule](#mockrule)
  - [mock.MatchRule](#mockmatchrule)
  - [ratelimiter.Policy](#ratelimiterpolicy)
@@ -180,6 +184,55 @@ pools:
 | serverError | Server-side network error |
 | failureCode | Resp failure code matches failureCodes set in poolSpec |
 
+## WebSocketProxy
+
+The WebSocketProxy filter is a proxy of the websocket backend service.
+
+Below is one of the simplest WebSocketProxy configurations, it forwards
+the websocket connection to `ws:https://127.0.0.1:9095`.
+
+```yaml
+kind: WebSocketProxy
+name: proxy-example-1
+pools:
+- servers:
+ - url: ws:https://127.0.0.1:9095
+```
+
+Same as the `Proxy` filter:
+* a `filter` can be configured on a pool.
+* the servers of a pool can be dynamically configured via service discovery.
+* When there are multiple servers in a pool, the pool can do a load balance
+ between them.
+
+Note, when routing traffic to a pipeline with a `WebSocketProxy`, the
+`HTTPServer` must set the corresponding `clientMaxBodySize` to `-1`, as
+below:
+
+```yaml
+name: demo-server
+kind: HTTPServer
+port: 8080
+rules:
+- paths:
+ path: /ws
+ clientMaxBodySize: -1 # REQUIRED!
+ backend: websocket-pipeline
+```
+
+### Configuration
+| Name | Type | Description | Required |
+| ---- | ---- | ----------- | -------- |
+| pools | [websocketproxy.WebSocketServerPoolSpec](#websocketproxywebsocketserverpoolspec) | The pool without `filter` is considered the main pool, other pools with `filter` are considered candidate pools, and a `Proxy` must contain exactly one main pool. When `WebSocketProxy` gets a request, it first goes through the candidate pools, and if it matches one of the pool's filter, servers of this pool handle the connection, otherwise, it is passed to the main pool. | Yes |
+| defaultOrigin | string | Easegress need to set the `Origin` header when connecting to the backend service, if the client request does not have this header, this value is used. | No |
+
+### Results
+
+| Value | Description |
+| ------------- | -------------------------------------------------------|
+| internalError | Encounters an internal error |
+| clientError | Client-side network error |
+
 ## CORSAdaptor
 
 The CORSAdaptor handles the [CORS](https://en.wikipedia.org/wiki/Cross-origin_resource_sharing) preflight, simple and not so simple request for the backend service.
@@ -945,7 +998,7 @@ Rules to revise request header.
 
 | Name | Type | Description | Required |
 | ------ | -------- | ------------------------------------------------------------------------------------------------------------ | -------- |
-| url | string | Address of the server. The address should start with `http:https://` or `https://`, followed by the hostname or IP address of the server, and then optionally followed by `:{port number}`, for example: `https://www.megaease.com`, `http:https://10.10.10.10:8080`. When host name is used, the `Host` of a request sent to this server is always the hostname of the server, and therefore using a [RequestAdaptor](#requestadaptor) in the pipeline to modify it will not be possible; when IP address is used, the `Host` is the same as the original request, that can be modified by a [RequestAdaptor](#requestadaptor). See also `KeepHost`. | Yes |
+| url | string | Address of the server. The address should start with `http:https://` or `https://` (when used in the `WebSocketProxy`, it can also start with `ws:https://` and `wss:https://`), followed by the hostname or IP address of the server, and then optionally followed by `:{port number}`, for example: `https://www.megaease.com`, `http:https://10.10.10.10:8080`. When host name is used, the `Host` of a request sent to this server is always the hostname of the server, and therefore using a [RequestAdaptor](#requestadaptor) in the pipeline to modify it will not be possible; when IP address is used, the `Host` is the same as the original request, that can be modified by a [RequestAdaptor](#requestadaptor). See also `KeepHost`. | Yes |
 | tags | []string | Tags of this server, refer `serverTags` in [proxy.PoolSpec](#proxyPoolSpec) | No |
 | weight | int | When load balance policy is `weightedRandom`, this value is used to calculate the possibility of this server | No |
 | keepHost | bool | If true, the `Host` is the same as the original request, no matter what is the value of `url`. Default value is `false`. | No |
@@ -1027,6 +1080,17 @@ The relationship between `methods` and `url` is `AND`.
 | keyBase64 | string | Base64 encoded key | Yes |
 | rootCertBase64 | string | Base64 encoded root certificate | Yes |
 
+### websocketproxy.WebSocketServerPoolSpec
+
+| Name | Type | Description | Required |
+| --------------- | -------------------------------------- | ------------------------------------------------------------------------------------------------------------ | -------- |
+| serverTags | []string | Server selector tags, only servers have tags in this array are included in this pool | No |
+| servers | [][proxy.Server](#proxyServer) | An array of static servers. If omitted, `serviceName` and `serviceRegistry` must be provided, and vice versa | No |
+| serviceName | string | This option and `serviceRegistry` are for dynamic server discovery | No |
+| serviceRegistry | string | This option and `serviceName` are for dynamic server discovery | No |
+| loadBalance | [proxy.LoadBalance](#proxyLoadBalanceSpec) | Load balance options | Yes |
+| filter | [proxy.RequestMatcherSpec](#proxyrequestmatcherspec) | Filter options for candidate pools | No |
+
 ### mock.Rule
 
 | Name | Type | Description | Required |

diff --git a/go.mod b/go.mod
@@ -15,7 +15,6 @@ require (
  github.com/goccy/go-json v0.9.6
  github.com/golang-jwt/jwt v3.2.1+incompatible
  github.com/google/uuid v1.3.0
- github.com/gorilla/websocket v1.5.0
  github.com/hashicorp/consul/api v1.14.0
  github.com/hashicorp/golang-lru v0.5.4
  github.com/invopop/yaml v0.2.0
@@ -55,7 +54,7 @@ require (
  go.etcd.io/etcd/server/v3 v3.5.4
  go.uber.org/zap v1.23.0
  golang.org/x/crypto v0.0.0-20220826181053-bd7e27e6170d
- golang.org/x/net v0.0.0-20220809184613-07c6da5e1ced
+ golang.org/x/net v0.0.0-20220919232410-f2f64ebce3c1
  golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4
  golang.org/x/sys v0.0.0-20220728004956-3c1f35247d10
  k8s.io/api v0.24.1
@@ -140,6 +139,7 @@ require (
  github.com/google/go-querystring v1.1.0 // indirect
  github.com/google/gofuzz v1.2.0 // indirect
  github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 // indirect
+ github.com/gorilla/websocket v1.5.0 // indirect
  github.com/gregjones/httpcache v0.0.0-20190611155906-901d90724c79 // indirect
  github.com/grpc-ecosystem/go-grpc-middleware v1.3.0 // indirect
  github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0 // indirect

diff --git a/go.sum b/go.sum
@@ -1135,8 +1135,9 @@ golang.org/x/net v0.0.0-20220412020605-290c469a71a5/go.mod h1:CfG3xpIq0wQ8r1q4Su
 golang.org/x/net v0.0.0-20220425223048-2871e0cb64e4/go.mod h1:CfG3xpIq0wQ8r1q4Su4UZFWDARRcnwPjda9FqA0JpMk=
 golang.org/x/net v0.0.0-20220520000938-2e3eb7b945c2/go.mod h1:CfG3xpIq0wQ8r1q4Su4UZFWDARRcnwPjda9FqA0JpMk=
 golang.org/x/net v0.0.0-20220725212005-46097bf591d3/go.mod h1:AaygXjzTFtRAg2ttMY5RMuhpJ3cNnI0XpyFJD1iQRSM=
-golang.org/x/net v0.0.0-20220809184613-07c6da5e1ced h1:3dYNDff0VT5xj+mbj2XucFst9WKk6PdGOrb9n+SbIvw=
 golang.org/x/net v0.0.0-20220809184613-07c6da5e1ced/go.mod h1:YDH+HFinaLZZlnHAfSS6ZXJJ9M9t4Dl22yv3iI2vPwk=
+golang.org/x/net v0.0.0-20220919232410-f2f64ebce3c1 h1:TWZxd/th7FbRSMret2MVQdlI8uT49QEtwZdvJrxjEHU=
+golang.org/x/net v0.0.0-20220919232410-f2f64ebce3c1/go.mod h1:YDH+HFinaLZZlnHAfSS6ZXJJ9M9t4Dl22yv3iI2vPwk=
 golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
 golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
 golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=