Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

THREESCALE-10224 CVE-2023-44487 http/2 rapid reset #1417

Merged
merged 6 commits into from
Nov 2, 2023

Conversation

eguzki
Copy link
Member

@eguzki eguzki commented Oct 25, 2023

What

Fixes https://issues.redhat.com/browse/THREESCALE-10224

CVE-2023-44487 (HTTP/2 Rapid Reset) fix

The actual fix is two-fold:

  • Bumping openresty to 1.19.3-23. The upgrade brings the fix from nginx https://hg.nginx.org/nginx/rev/cdda286c0f1b
  • Enable openresty [lua_check_client_abort|https://github.com/openresty/lua-nginx-module#lua_check_client_abort] to check for premature client connection abortion.

TL;DR the Rapid reset sec threat is about doing lot's of requests with rapid reset right after sending the request so the [http2_max_concurrent_streams|https://nginx.org/en/docs/http/ngx_http_v2_module.html#http2_max_concurrent_streams] threshold is never reached.

With the openresty [lua_check_client_abort|https://github.com/openresty/lua-nginx-module#lua_check_client_abort] disabled, apicast was not honoring the resets and the http2_max_concurrent_streams threshold (128 by default in nginx) was hit. Even though the client was sending the reset. So, APICast was protected to the rapid reset sec threat. However that happened because HTTP2 protocol was not correctly implemented. APICast was sending data even after receiving the reset in the closed stream.

Thus, this PR enables lua_check_client_abort, which makes APIcast to respect the resets, thus making it vulnerable to the rapid reset sec threat. But also more compliant with HTTP2. The protection against the rapid reset sec threat comes from the nginx fix https://hg.nginx.org/nginx/rev/cdda286c0f1b

The client used to exploit the rapid reset vulnerability: https://github.com/eguzki/nice-cve-poc/tree/ssl-optional

Related links:

Verification steps

  • Checkout this branch
git checkout origin/cve-2023-44487-rapid-reset
  • Build docker image apicast-cve-2023-44487
make runtime-image IMAGE_NAME=apicast-cve-2023-44487
  • Run dummy HTTP/2 echo api
docker run --name echoserver --rm -p 8002-8003:8002-8003 kalmhq/echoserver:latest
  • Create gateway certs for example.com (inside the container)
cd /tmp/apicast-test && cd /tmp/apicast-test
openssl genrsa -out rootCA.key 2048
openssl req -batch -new -x509 -nodes -key rootCA.key -sha256 -days 1024 -out rootCA.pem
openssl req -subj '/CN=example.com'  -newkey rsa:4096 -nodes -sha256 -days 3650 -keyout example.com.key -out example.com.csr
chmod +r example.com.key
openssl x509 -req -in example.com.csr -CA rootCA.pem -CAkey rootCA.key -CAcreateserial -out example.com.crt -days 500 -sha256
  • Create apicast configuration
ECHOSERVER_IP=$(docker inspect echoserver | yq e -P '.[0].NetworkSettings.Networks.bridge.IPAddress' -)
❯ cat <<EOF >config.json
{
  "services": [
    {
      "backend_version": "1",
      "proxy": {
        "hosts": [
          "example.com"
        ],
        "api_backend": "https://${ECHOSERVER_IP}:8003",
        "backend": {
          "endpoint": "https://127.0.0.1:8081",
          "host": "backend"
        },
        "proxy_rules": [
          {
            "http_method": "GET",
            "pattern": "/",
            "metric_system_name": "hits",
            "delta": 1,
            "parameters": [],
            "querystring_parameters": {}
          }
        ],
        "policy_chain": [
          {
                "name": "default_credentials",
                "version": "builtin",
                "configuration": {
                  "auth_type": "user_key",
                  "user_key": "some_dummy_key"
                }
          },
          {
            "name": "apicast.policy.apicast"
          }
        ]
      }
    }
  ]
}
EOF
  • Run apicast locally. the output will be filtered, looking for the log client sent too many streams at once while processing HTTP/2 connection. That log is generated by the fix from nginx https://hg.nginx.org/nginx/rev/cdda286c0f1b
docker run -t --name apicast --rm -v $PWD:/var/run/secrets/apicast -v $PWD/config.json:/opt/app/config.json:ro --env THREESCALE_CONFIG_FILE=/opt/app/config.json --env APICAST_LOG_LEVEL=info --env APICAST_CONFIGURATION_LOADER=lazy --env THREESCALE_DEPLOYMENT_ENV=staging --env APICAST_HTTPS_PORT=8443 --env APICAST_HTTPS_CERTIFICATE=/var/run/secrets/apicast/example.com.crt --env APICAST_HTTPS_CERTIFICATE_KEY=/var/run/secrets/apicast/example.com.key --env APICAST_WORKERS=1 --env APICAST_CONFIGURATION_CACHE=0 apicast-cve-2023-44487 | grep "client sent too many streams at once while processing HTTP/2 connection"
APICAST_IP=$(docker inspect apicast | yq e -P '.[0].NetworkSettings.Networks.bridge.IPAddress' -)

# Just one request to test e2e works
curl --http2 --resolve example.com:8443:${APICAST_IP} -v --cacert rootCA.pem "https://example.com:8443"

# Run the traffic tool. The following run sends 300 request/reset frames. Depending on your resources, more or less is required. At least 2 * max_concurrent_streams (which is 128 in nginx by default)
docker run --rm -t  quay.io/eastizle/cve-2023-44487:v0.0.2 --host example.com --hits 300  https://${APICAST_IP}:8443

The apicast log line should show

2023/10/31 17:21:03 [info] 19#19: *404 client sent too many streams at once while processing HTTP/2 connection, client: 172.17.0.4, server: 0.0.0.0:8443

@eguzki eguzki requested a review from a team as a code owner October 25, 2023 14:05
Copy link
Contributor

@tkan145 tkan145 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Look good to me. +1

Verification

  • Checkout this branch
git checkout origin/cve-2023-44487-rapid-reset
  • Run docker dev image
make development
  • Check the running image
 ▲ lib/3scale/APIcast docker ps
CONTAINER ID   IMAGE                                           COMMAND                  CREATED         STATUS         PORTS      NAMES
487d71673b69   quay.io/3scale/apicast-ci:openresty-1.19.3-23   "cat"                    8 minutes ago   Up 8 minutes   8080/tcp   apicast_build_0-development-1
52b7c5009bc4   redis                                           "docker-entrypoint.s…"   44 hours ago    Up 8 minutes   6379/tcp   apicast_build_0-redis-1

@eguzki eguzki merged commit 292d738 into master Nov 2, 2023
11 checks passed
@eguzki eguzki deleted the cve-2023-44487-rapid-reset branch November 2, 2023 15:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants