chore(self-hosted): Add self-hosted e2e test action

[hubertdeng123]

We've done a lot of work this quarter to improve the self-hosted e2e test suite, and want to turn this back on in snuba.

More here: https://www.notion.so/sentry/Getting-self-hosted-e2e-tests-back-on-pre-commit-8a3e91cfe7fa475689f343c4d41eca31?pvs=4

Tests should no longer be as flaky as before, which is why it was removed in #4660

[volokluev]

it looks pretty flaky to me just looking at the commit history 👀

[hubertdeng123]

it looks pretty flaky to me just looking at the commit history 👀

Sorry, will need to wait for getsentry/self-hosted#3002 to go in first. That should solve the red X's

[hubertdeng123]

@volokluev I've added the continue-on-error: true to monitor the job for a week before flipping it on as a required check.

[codecov]

Test Failures Detected: Due to failing tests, we cannot provide coverage reports at this time.

❌ Failed Test Results:

Completed 624 tests with 1 failed, 622 passed and 1 skipped.

View the full list of failed tests

Test Description

Failure message

Testsuite: pytest Test name: tests.clickhouse.optimize.test_optimize_scheduler::test_get_next_schedule[non parallel] Envs: - default

Traceback (most recent call last): File ".../clickhouse/optimize/test_optimize_scheduler.py", line 254, in test_get_next_schedule assert optimize_scheduler.get_next_schedule(partitions) == expected AssertionError: assert OptimizationSchedule(partitions=[["(90,'2022-06-27')",\n "(90,'2022-06-20')",\n "(90,'2022-06-13')",\n "(90,'2022-06-08')"]],\n cutoff_time=datetime.datetime(2024, 5, 1, 0, 0),\n start_time_jitter_minutes=None) == OptimizationSchedule(partitions=[["(90,'2022-06-27')",\n "(90,'2022-06-20')",\n "(90,'2022-06-13')",\n "(90,'2022-06-08')"]],\n cutoff_time=datetime.datetime(2024, 4, 30, 0, 0),\n start_time_jitter_minutes=None) Matching attributes: ['partitions', 'start_time_jitter_minutes'] Differing attributes: ['cutoff_time'] Drill down into differing attribute cutoff_time: cutoff_time: datetime.datetime(2024, 5, 1, 0, 0) != datetime.datetime(2024, 4, 30, 0, 0)