document limitation of stream req/resp (#671)

doc/reference/controllers.md

@@ -9,10 +9,10 @@
  - [Pipeline](#pipeline)
  - [StatusSyncController](#statussynccontroller)
  - [Business Controllers](#business-controllers)
+ - [GlobalFilter](#globalfilter)
  - [EaseMonitorMetrics](#easemonitormetrics)
  - [FaaSController](#faascontroller)
  - [IngressController](#ingresscontroller)
- - [MeshController](#meshcontroller)
  - [ConsulServiceRegistry](#consulserviceregistry)
  - [EtcdServiceRegistry](#etcdserviceregistry)
  - [EurekaServiceRegistry](#eurekaserviceregistry)
@@ -26,12 +26,15 @@
  - [httpserver.Rule](#httpserverrule)
  - [httpserver.Path](#httpserverpath)
  - [httpserver.Header](#httpserverheader)
- - [pipeline.FlowNode](#pipeline.FlowNode)
- - [filters.Filter](#filtersFilter)
+ - [pipeline.Spec](#pipelinespec)
+ - [pipeline.FlowNode](#pipelineflownode)
+ - [filters.Filter](#filtersfilter)
  - [easemonitormetrics.Kafka](#easemonitormetricskafka)
  - [nacos.ServerSpec](#nacosserverspec)
  - [autocertmanager.DomainSpec](#autocertmanagerdomainspec)
- - [resilience.Policy](#resiliencePolicy)
+ - [resilience.Policy](#resiliencepolicy)
+ - [Retry Policy](#retry-policy)
+ - [Circuit Break Policy](#circuit-break-policy)
 
 As the [architecture diagram](../imgs/architecture.png) shows, the controller is the core entity to control kinds of working. There are two kinds of controllers overall:
 
@@ -101,7 +104,7 @@ rules:
 | ipFilter | [ipfilter.Spec](#ipfilterSpec) | IP Filter for all traffic under the server | No |
 | rules | [httpserver.Rule](#httpserverRule) | Router rules | No |
 | autoCert | bool | Do HTTP certification automatically | No | 
-| clientMaxBodySize | int64 | Max size of request body. the default value is 4MB. Requests with a body larger than this option are discarded. When this option is set to `-1`, Easegress takes the request body as a stream and the body can be any size, but some features are not possible in this case. | No | 
+| clientMaxBodySize | int64 | Max size of request body. the default value is 4MB. Requests with a body larger than this option are discarded. When this option is set to `-1`, Easegress takes the request body as a stream and the body can be any size, but some features are not possible in this case, please refer [Stream](./stream.md) for more information. | No | 
 | caCertBase64 | string | Define the root certificate authorities that servers use if required to verify a client certificate by the policy in TLS Client Authentication. | No |
 | globalFilter | string | Name of [GlobalFilter](#globalfilter) for all backends | No | 
 
@@ -229,13 +232,6 @@ filters:
 ```
 In this case, we give second `proxy` alias `proxy2`, so request is invalid, it jumps to second proxy. 
 
-#### pipeline.Spec 
-| Name | Type | Description | Required | 
-|------|------|-------------|----------|
-| flow | [pipeline.FlowNode](#pipelineFlowNode) | Flow of pipeline | No |
-| filters | [][filters.Filter](#filters.Filter) | Filter definitions of pipeline | Yes |
-| resilience | [][resilience.Policy](#resiliencePolicy) | Resilience policy for backend filters | No | 
-
 ### StatusSyncController
 
 No config.
@@ -502,7 +498,7 @@ domains:
 | methods | []string | Methods to match, empty means to allow all methods | No |
 | headers | [][httpserver.Header](#httpserverHeader) | Headers to match (the requests matching headers won't be put into cache) | No |
 | backend | string | backend name (pipeline name in static config, service name in mesh) | Yes |
-| clientMaxBodySize | int64 | Max size of request body. the default value is 4MB. Requests with a body larger than this option are discarded. When this option is set to `-1`, Easegress takes the request body as a stream and the body can be any size, but some features are not possible in this case. | No | 
+| clientMaxBodySize | int64 | Max size of request body, will use the option of the HTTP server if not set. the default value is 4MB. Requests with a body larger than this option are discarded. When this option is set to `-1`, Easegress takes the request body as a stream and the body can be any size, but some features are not possible in this case, please refer [Stream](./stream.md) for more information. | No | 
 | matchAllHeader | bool | Match all headers that are defined in headers, default is `false`. | No |
 
 
@@ -516,6 +512,12 @@ There must be at least one of `values` and `regexp`.
 | values | []string | Header values to match | No |
 | regexp | string | Header value in regular expression to match | No |
 
+### pipeline.Spec 
+| Name | Type | Description | Required | 
+|------|------|-------------|----------|
+| flow | [pipeline.FlowNode](#pipelineFlowNode) | Flow of pipeline | No |
+| filters | [][filters.Filter](#filters.Filter) | Filter definitions of pipeline | Yes |
+| resilience | [][resilience.Policy](#resiliencePolicy) | Resilience policy for backend filters | No | 
 
 ### pipeline.FlowNode
 

doc/reference/filters.md

@@ -165,7 +165,7 @@ pools:
 | mtls | [proxy.MTLS](#proxymtls) | mTLS configuration | No |
 | maxIdleConns | int | Controls the maximum number of idle (keep-alive) connections across all hosts. Default is 10240 | No |
 | maxIdleConnsPerHost | int | Controls the maximum idle (keep-alive) connections to keep per-host. Default is 1024 | No |
-| serverMaxBodySize | int64 | Max size of response body. the default value is 4MB. Responses with a body larger than this option are discarded. When this option is set to `-1`, Easegress takes the response body as a stream and the body can be any size, but some features are not possible in this case. | No |
+| serverMaxBodySize | int64 | Max size of response body. the default value is 4MB. Responses with a body larger than this option are discarded. When this option is set to `-1`, Easegress takes the response body as a stream and the body can be any size, but some features are not possible in this case, please refer [Stream](./stream.md) for more information. | No |
 
 ### Results
 
@@ -864,7 +864,7 @@ Rules to revise request header.
 | loadBalance | [proxy.LoadBalance](#proxyLoadBalanceSpec) | Load balance options | Yes |
 | memoryCache | [proxy.MemoryCacheSpec](#proxymemorycachespec) | Options for response caching | No |
 | filter | [proxy.RequestMatcherSpec](#proxyrequestmatcherspec) | Filter options for candidate pools | No |
-| serverMaxBodySize | int64 | Request max body size | No | 
+| serverMaxBodySize | int64 | Max size of response body, will use the option of the Proxy if not set. Responses with a body larger than this option are discarded. When this option is set to `-1`, Easegress takes the response body as a stream and the body can be any size, but some features are not possible in this case, please refer [Stream](./stream.md) for more information. | No |
 | timeout | string | Request calceled when timeout | No | 
 | retryPolicy | string | Retry policy name | No |
 | circuitBreakPolicy | string | Circuit break policy name | No | 

doc/reference/stream.md

@@ -0,0 +1,35 @@
+# Stream
+
+Most Easegress traffic is message-based, but Easegress v2 pipeline is
+protocol independent, that's it supports stream-based traffic like TCP, and
+there's stream traffic even in HTTP.
+
+Another new feature in Easegress v2 is multiple requests/responses support,
+this requires the payload of a request/response can be read more than once,
+for message-based traffic, this is simple and easy to do, as Easegress can
+read the full message payload into memory. But for stream-based traffic,
+this is impossible, as the payload may require too much memory, and/or take
+too much time to read it into memory.
+
+To resolve the above issue, Easegress allow user or developer to configure
+whether a request/response is a stream, for example:
+
+* We can set `clientMaxBodySize` of an HTTP server to a negative value to
+ tell Easegress the request is a stream, and not a stream otherwise. Please
+ refer [HTTP Server](./controllers.md#httpserver) for more information.
+* We can set `serverMaxBodySize` of a `Proxy` filter to a negative value to
+ tell Easegress the response is a stream, and not a stream otherwise. Please
+ refer [Proxy](./filters.md#proxy) for more information.
+
+As we have mentioned above, the payload of a stream-based request/response
+cannot be read once, so some features are not possible for these
+requests/responses, including:
+
+* In the `template` of `RequestBuilder` or `ResponseBuilder`, you cannot
+ access the payload(in HTTP, the body) of an existing stream-based
+ request/response, while it is fine to access other information of the
+ request/response.
+* Stream-based request/response cannot be cached by `Proxy`.
+* The `HeaderToJSON` filter does not support stream-based requests/responses.
+* You cannot access the payload of stream-based request/response in a
+ `WasmHost` filter.