feat: Sync fork (#8)

* feat: provide csrf settings information for sentry config (getsentry#2762) * feat: provide csrf settings information for sentry config * chore: trim trailing whitespace * build(deps): bump actions/setup-python from 4 to 5 (getsentry#2644) Bumps [actions/setup-python](https://github.com/actions/setup-python) from 4 to 5. - [Release notes](https://github.com/actions/setup-python/releases) - [Commits](actions/setup-python@v4...v5) --- updated-dependencies: - dependency-name: actions/setup-python dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Hubert Deng <[email protected]> * Fix groupedmessage indexing error (getsentry#2777) * fix groupedmessage indexing error * Check memcached backend in Django (getsentry#2778) Bail if using old memcached backend * release: 24.1.2 * build: Set master version to nightly #skip-changelog * fix: DB migration script (getsentry#2779) * use different approach to wait for postgres server * Tweak postgres indexing fix (getsentry#2792) * tweak postgres indexing fix * add exists constraint * build(deps): bump pre-commit/action from 3.0.0 to 3.0.1 (getsentry#2788) Bumps [pre-commit/action](https://github.com/pre-commit/action) from 3.0.0 to 3.0.1. - [Release notes](https://github.com/pre-commit/action/releases) - [Commits](pre-commit/action@v3.0.0...v3.0.1) --- updated-dependencies: - dependency-name: pre-commit/action dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump nginx version (getsentry#2797) bump nginx version * release: 24.2.0 * build: Set master version to nightly #skip-changelog * use rust consumers in self-hosted * revert changes in 3067683 * chore: Use django ORM to perform sql commands (getsentry#2827) * use django orm * chore: provide clearer csrf url example (getsentry#2833) * Upgrade to FSL-1.1 (getsentry#2835) * simplify if for open-ai-suggestion (getsentry#2732) * simplify if for open-ai-suggestion * Add snuba rust consumers (getsentry#2831) * add snuba rust consumers * Enable proxy buffering in nginx (getsentry#2844) Enable proxy buffering Setting the proxy_buffering directive to “off” is a common mistake because it can cause performance issues and unexpected behavior in NGINX. When proxy buffering is disabled, NGINX receives a response from the proxied server and immediately sends it to the client without storing it in a buffer. This can cause problems if the response is large or if the connection between NGINX and the client is slow, because it can result in increased memory usage and potentially lead to request timeouts. To avoid this mistake, it is recommended to always enable proxy buffering in NGINX. * deps: bump maxmind/geoipupdate to 6.1.0 (getsentry#2859) https://github.com/maxmind/geoipupdate/releases/tag/v6.1.0 * increase postgres max_connections above 100 connections (getsentry#2740) * use default value and env for postgres max_connections * Integration test improvements (getsentry#2858) * integration test improvements * feat(spans): Ingest spans (getsentry#2861) * release: 24.3.0 * build: Set master version to nightly #skip-changelog * Remove duplicate feature flags (getsentry#2899) * feat: run outcomes-billing consumer (getsentry#2909) * Integration tests in python (getsentry#2892) * integration tests in python * Fix defunct java processes (getsentry#2914) revert kafka healthcheck change * Port backup tests to python (getsentry#2907) * port backup tests to python * feat(clickhouse): Added max_suspicious_broken_parts to the config.xml (getsentry#2853) * feat(clickhouse): Added max_suspicious_broken_parts to the config.xml * refactor(clickhouse): Set default max_suspicious_bronken_parts and Issue reference --------- Co-authored-by: Hubert Deng <[email protected]> * Write Customization tests in python (getsentry#2918) * port everything integration test related to python * Bump ubuntu version for tests (getsentry#2923) * bump ubuntu version used for testing * get rid of codecov cli dependency * fix(spans): Adds organizations:standalone-span-ingestion flag to default config (getsentry#2936) Adds organizations:standalone-span-ingestion flag to default config * feat: adds group attributes consumer (getsentry#2927) adds group attributes consumer * Use python for e2e tests (getsentry#2953) * bump e2e action commit sha * release: 24.4.0 * build: Set master version to nightly #skip-changelog * Port last integration tests to python (getsentry#2966) * port custom ca cert test to python * Add example to docker compose version in problem report (getsentry#2959) * Use docker compose exec to create additional kafka topics (getsentry#2904) * chore(deps): bump memcached and redis to latest patch versions (getsentry#2973) * release: 24.4.1 * build: Set master version to nightly #skip-changelog * Add workstation configuration (getsentry#2968) * Add workstation configuration These are prebuilt docker images for spinning up a local self-hosted image on the Google Cloud Workstation project. While primarily intended for internal development at Sentry, in theory these can be used by anyone with GCWS project to create a fresh workstation for developing self-hosted via a remote VSCode connection. Users who have GCWS properly configured will be able to use the forthcoming `workstations ...` command in the `sentry` dev CLI to create, manage, and destroy one-off or long-lived workstations in either the pre-install or post-install configuration. Note that the `sentry workstations ...` CLI has not yet landed in the `sentry` repo - those changes are coming soon! Issue: getsentry/team-ospo#240 * Fix shfmt complaints * Upgrade postgres to 14.11 (getsentry#2975) chore(deps): bump postgres to latest 14 alpine version * Bump docker compose version in CI (getsentry#2980) * only rerun tests on v2.0.1 * change from http error to request error * use 3 retries like before * fix: backport changes --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Reinaldy Rafli <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Hubert Deng <[email protected]> Co-authored-by: Chad Whitacre <[email protected]> Co-authored-by: getsentry-bot <[email protected]> Co-authored-by: getsentry-bot <[email protected]> Co-authored-by: hubertdeng123 <[email protected]> Co-authored-by: Iven Schlenther <[email protected]> Co-authored-by: RexTim <[email protected]> Co-authored-by: Victor <[email protected]> Co-authored-by: Erfan <[email protected]> Co-authored-by: Pierre Massat <[email protected]> Co-authored-by: Jann Kleen <[email protected]> Co-authored-by: Lyn Nagara <[email protected]> Co-authored-by: edwardgou-sentry <[email protected]> Co-authored-by: Stephen Cefali <[email protected]> Co-authored-by: Edgar Sanchez <[email protected]> Co-authored-by: Steffen Zieger <[email protected]> Co-authored-by: Matthew T <[email protected]> Co-authored-by: Alex Zaslavsky <[email protected]>
KingsGambitLab · Apr 20, 2024 · 480ee55 · 480ee55
1 parent 9b65a46
commit 480ee55
Show file tree

Hide file tree

Showing 48 changed files with 1,119 additions and 406 deletions.
diff --git a/.env b/.env
@@ -14,3 +14,6 @@ WAL2JSON_VERSION=latest
 HEALTHCHECK_INTERVAL=30s
 HEALTHCHECK_TIMEOUT=1m30s
 HEALTHCHECK_RETRIES=10
+# Caution: Raising max connections of postgres increases CPU and RAM usage
+# see https://github.com/getsentry/self-hosted/pull/2740 for more information
+POSTGRES_MAX_CONNECTIONS=100
diff --git a/.github/ISSUE_TEMPLATE/problem-report.yml b/.github/ISSUE_TEMPLATE/problem-report.yml
@@ -36,6 +36,7 @@ body:
  placeholder: 2.6.0 ← should look like this (docker compose version)
  description: |
  What version of docker compose are you using to run self-hosted?
+ e.g: (docker compose version)
  validations:
  required: true
  - type: textarea

diff --git a/.gitignore b/.gitignore
@@ -101,3 +101,6 @@ postgres/wal2json
 # integration testing
 _integration-test/custom-ca-roots/nginx/*
 sentry/test-custom-ca-roots.py
+
+# OSX minutia
+.DS_Store
diff --git a/.platform/confighooks/prebuild/01_prebuild.sh b/.platform/confighooks/prebuild/01_prebuild.sh
@@ -2,7 +2,7 @@
 
 set -e
 /opt/elasticbeanstalk/bin/get-config environment | jq -r 'to_entries | .[] | "\(.key)=\"\(.value)\""' > .env
-printf "SENTRY_IMAGE=getsentry/sentry:23.12.1" >> .env
+printf "SENTRY_IMAGE=getsentry/sentry:24.4.1" >> .env
 set -a
 source .env
 set +a

diff --git a/.platform/confighooks/predeploy/01_loadenv.sh b/.platform/confighooks/predeploy/01_loadenv.sh
@@ -2,7 +2,7 @@
 
 set -e
 /opt/elasticbeanstalk/bin/get-config environment | jq -r 'to_entries | .[] | "\(.key)=\"\(.value)\""' > .env
-printf "SENTRY_IMAGE=getsentry/sentry:23.12.1" >> .env
+printf "SENTRY_IMAGE=getsentry/sentry:24.4.1" >> .env
 set -a
 source .env
 set +a

diff --git a/.platform/hooks/prebuild/01_prebuild.sh b/.platform/hooks/prebuild/01_prebuild.sh
@@ -2,7 +2,7 @@
 
 set -e
 /opt/elasticbeanstalk/bin/get-config environment | jq -r 'to_entries | .[] | "\(.key)=\"\(.value)\""' > .env
-printf "SENTRY_IMAGE=getsentry/sentry:23.12.1" >> .env
+printf "SENTRY_IMAGE=getsentry/sentry:24.4.1" >> .env
 set -a
 source .env
 set +a

diff --git a/.platform/hooks/predeploy/01_loadenv.sh b/.platform/hooks/predeploy/01_loadenv.sh
@@ -2,7 +2,7 @@
 
 set -e
 /opt/elasticbeanstalk/bin/get-config environment | jq -r 'to_entries | .[] | "\(.key)=\"\(.value)\""' > .env
-printf "SENTRY_IMAGE=getsentry/sentry:23.12.1" >> .env
+printf "SENTRY_IMAGE=getsentry/sentry:24.4.1" >> .env
 set -a
 source .env
 set +a

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,5 +1,66 @@
 # Changelog
 
+## 24.4.1
+
+### Various fixes & improvements
+
+- chore(deps): bump memcached and redis to latest patch versions (#2973) by @mdtro
+- Use docker compose exec to create additional kafka topics (#2904) by @saz
+- Add example to docker compose version in problem report (#2959) by @edgariscoding
+- Port last integration tests to python (#2966) by @hubertdeng123
+
+## 24.4.0
+
+### Various fixes & improvements
+
+- Use python for e2e tests (#2953) by @hubertdeng123
+- feat: adds group attributes consumer (#2927) by @scefali
+- fix(spans): Adds organizations:standalone-span-ingestion flag to default config (#2936) by @edwardgou-sentry
+- Bump ubuntu version for tests (#2923) by @hubertdeng123
+- Write Customization tests in python (#2918) by @hubertdeng123
+- feat(clickhouse): Added max_suspicious_broken_parts to the config.xml (#2853) by @victorelec14
+- Port backup tests to python (#2907) by @hubertdeng123
+- Fix defunct java processes (#2914) by @hubertdeng123
+- Integration tests in python (#2892) by @hubertdeng123
+- feat: run outcomes-billing consumer (#2909) by @lynnagara
+- Remove duplicate feature flags (#2899) by @JannKleen
+
+## 24.3.0
+
+### Various fixes & improvements
+
+- feat(spans): Ingest spans (#2861) by @phacops
+- Integration test improvements (#2858) by @hubertdeng123
+- increase postgres max_connections above 100 connections (#2740) by @erfantkerfan
+- deps: bump maxmind/geoipupdate to 6.1.0 (#2859) by @victorelec14
+- Enable proxy buffering in nginx (#2844) by @RexTim
+- Add snuba rust consumers (#2831) by @hubertdeng123
+- simplify if for open-ai-suggestion (#2732) by @LvckyAPI
+- Upgrade to FSL-1.1 (#2835) by @chadwhitacre
+- chore: provide clearer csrf url example (#2833) by @aldy505
+- chore: Use django ORM to perform sql commands (#2827) by @hubertdeng123
+- revert changes in 3067683f6c0e1c6dd9ceb72cb5155c1dbf3bf501 (#2829) by @hubertdeng123
+- use rust consumers in self-hosted (3067683f) by @hubertdeng123
+
+## 24.2.0
+
+### Various fixes & improvements
+
+- Bump nginx version (#2797) by @hubertdeng123
+- build(deps): bump pre-commit/action from 3.0.0 to 3.0.1 (#2788) by @dependabot
+- Tweak postgres indexing fix (#2792) by @hubertdeng123
+- fix: DB migration script (#2779) by @hubertdeng123
+
+## 24.1.2
+
+### Various fixes & improvements
+
+- Check memcached backend in Django (#2778) by @chadwhitacre
+- Fix groupedmessage indexing error (#2777) by @hubertdeng123
+- build(deps): bump actions/setup-python from 4 to 5 (#2644) by @dependabot
+- feat: provide csrf settings information for sentry config (#2762) by @aldy505
+- Fix apt config generation when http_proxy is set (#2725) (#2734) by @lemrouch
+
 ## 24.1.1
 
 ### Various fixes & improvements

diff --git a/LICENSE.md b/LICENSE.md
@@ -1,12 +1,12 @@
-# Functional Source License, Version 1.0, Apache 2.0 Change License
+# Functional Source License, Version 1.1, Apache 2.0 Future License
 
 ## Abbreviation
 
-FSL-1.0-Apache-2.0
+FSL-1.1-Apache-2.0
 
 ## Notice
 
-Copyright 2016-2023 Functional Software, Inc. dba Sentry
+Copyright 2016-2024 Functional Software, Inc. dba Sentry
 
 ## Terms and Conditions
 
@@ -30,18 +30,15 @@ and redistribute the Software for any Permitted Purpose identified below.
 ### Permitted Purpose
 
 A Permitted Purpose is any purpose other than a Competing Use. A Competing Use
-means use of the Software in or for a commercial product or service that
-competes with the Software or any other product or service we offer using the
-Software as of the date we make the Software available.
+means making the Software available to others in a commercial product or
+service that:
 
-Competing Uses specifically include using the Software:
+1. substitutes for the Software;
 
-1. as a substitute for any of our products or services;
+2. substitutes for any other product or service we offer using the Software
+ that exists as of the date we make the Software available; or
 
-2. in a way that exposes the APIs of the Software; and
-
-3. in a product or service that offers the same or substantially similar
- functionality to the Software.
+3. offers the same or substantially similar functionality as the Software.
 
 Permitted Purposes specifically include using the Software:
 
@@ -51,9 +48,8 @@ Permitted Purposes specifically include using the Software:
 
 3. for non-commercial research; and
 
-4. in connection with software development services or managed services that
- you provide to a licensee using the Software in accordance with these Terms
-and Conditions.
+4. in connection with professional services that you provide to a licensee
+ using the Software in accordance with these Terms and Conditions.
 
 ### Patents
 
@@ -88,12 +84,13 @@ Except for displaying the License Details and identifying us as the origin of
 the Software, you have no right under these Terms and Conditions to use our
 trademarks, trade names, service marks or product names.
 
-## Change License
+## Grant of Future License
 
-On the second anniversary of the date we make the Software available, the
-Software will become available under the Apache 2.0 license. On that date, the
-Terms and Conditions above automatically terminate and the following terms
-become effective:
+We hereby irrevocably grant you an additional license to use the Software under
+the Apache License, Version 2.0 that is effective on the second anniversary of
+the date we make the Software available. On or after that date, you may use the
+Software under the Apache License, Version 2.0, in which case the following
+will apply:
 
 Licensed under the Apache License, Version 2.0 (the "License"); you may not use
 this file except in compliance with the License.

diff --git a/_integration-test/conftest.py b/_integration-test/conftest.py
@@ -0,0 +1,82 @@
+import os
+import subprocess
+import time
+
+import httpx
+import pytest
+
+SENTRY_CONFIG_PY = "sentry/sentry.conf.py"
+SENTRY_TEST_HOST = os.getenv("SENTRY_TEST_HOST", "https://localhost:9000")
+TEST_USER = "[email protected]"
+TEST_PASS = "test123TEST"
+TIMEOUT_SECONDS = 60
+
+
+def pytest_addoption(parser):
+ parser.addoption("--customizations", default="disabled")
+
+
+@pytest.fixture(scope="session", autouse=True)
+def configure_self_hosted_environment(request):
+ subprocess.run(
+ ["docker", "compose", "--ansi", "never", "up", "-d"],
+ check=True,
+ capture_output=True,
+ )
+ for i in range(TIMEOUT_SECONDS):
+ try:
+ response = httpx.get(SENTRY_TEST_HOST, follow_redirects=True)
+ except httpx.RequestError:
+ time.sleep(1)
+ else:
+ if response.status_code == 200:
+ break
+ else:
+ raise AssertionError("timeout waiting for self-hosted to come up")
+
+ if request.config.getoption("--customizations") == "enabled":
+ os.environ["TEST_CUSTOMIZATIONS"] = "enabled"
+ script_content = """\
+#!/bin/bash
+touch /created-by-enhance-image
+apt-get update
+apt-get install -y gcc libsasl2-dev python-dev libldap2-dev libssl-dev
+"""
+
+ with open("sentry/enhance-image.sh", "w") as script_file:
+ script_file.write(script_content)
+ # Set executable permissions for the shell script
+ os.chmod("sentry/enhance-image.sh", 0o755)
+
+ # Write content to the requirements.txt file
+ with open("sentry/requirements.txt", "w") as req_file:
+ req_file.write("python-ldap\n")
+ os.environ["MINIMIZE_DOWNTIME"] = "1"
+ subprocess.run(["./install.sh"], check=True)
+ # Create test user
+ subprocess.run(
+ [
+ "docker",
+ "compose",
+ "exec",
+ "-T",
+ "web",
+ "sentry",
+ "createuser",
+ "--force-update",
+ "--superuser",
+ "--email",
+ TEST_USER,
+ "--password",
+ TEST_PASS,
+ "--no-input",
+ ],
+ check=True,
+ text=True,
+ )
+
+
+@pytest.fixture()
+def setup_backup_restore_env_variables():
+ os.environ["SENTRY_DOCKER_IO_DIR"] = os.path.join(os.getcwd(), "sentry")
+ os.environ["SKIP_USER_CREATION"] = "1"
diff --git a/_integration-test/custom-ca-roots/setup.sh b/_integration-test/custom-ca-roots/setup.sh
diff --git a/_integration-test/custom-ca-roots/teardown.sh b/_integration-test/custom-ca-roots/teardown.sh
diff --git a/_integration-test/custom-ca-roots/test.py b/_integration-test/custom-ca-roots/test.py
@@ -1,15 +1,16 @@
 import unittest
+
 import requests
 
 
 class CustomCATests(unittest.TestCase):
  def test_valid_self_signed(self):
- self.assertEqual(requests.get("https://self.test").text, 'ok')
+ self.assertEqual(requests.get("https://self.test").text, "ok")
 
  def test_invalid_self_signed(self):
  with self.assertRaises(requests.exceptions.SSLError):
  requests.get("https://fail.test")
 
 
-if __name__ == '__main__':
+if __name__ == "__main__":
  unittest.main()
diff --git a/_integration-test/ensure-backup-restore-works.sh b/_integration-test/ensure-backup-restore-works.sh
diff --git a/_integration-test/ensure-customizations-not-present.sh b/_integration-test/ensure-customizations-not-present.sh