Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Integration test improvements #2858

Merged
merged 11 commits into from
Mar 7, 2024
Merged

Integration test improvements #2858

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
02ccedd
integration test improvements
hubertdeng123 Mar 6, 2024
07c059e
fix sentry send event
hubertdeng123 Mar 6, 2024
364d0d2
install self-hsoted as separate step in gh action
hubertdeng123 Mar 6, 2024
9dd9ac0
do not report self-hosted issues during install
hubertdeng123 Mar 6, 2024
c21576f
run jobs for compose v2.0.1 and v2.7.0
hubertdeng123 Mar 6, 2024
4308b2c
incorporate sentry dsn for testing into existing logic
hubertdeng123 Mar 6, 2024
b22f1eb
fix test type
hubertdeng123 Mar 7, 2024
9c4fbfc
remove retries for now
hubertdeng123 Mar 7, 2024
ffabc9f
comment onto using exec
hubertdeng123 Mar 7, 2024
2ced8fb
fix error reporting
hubertdeng123 Mar 7, 2024
7559e55
add error handling to test.sh
hubertdeng123 Mar 7, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 18 additions & 1 deletion .github/workflows/test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,8 @@ jobs:
strategy:
fail-fast: false
matrix:
test_type: ["initial-install", "customizations"]
compose_version: ["v2.0.1", "v2.7.0"]
include:
- compose_version: "v2.0.1"
compose_path: "/usr/local/lib/docker/cli-plugins"
Expand All @@ -82,8 +84,23 @@ jobs:
sudo curl -L https://github.com/docker/compose/releases/download/${{ matrix.compose_version }}/docker-compose-`uname -s`-`uname -m` -o "${{ matrix.compose_path }}/docker-compose"
sudo chmod +x "${{ matrix.compose_path }}/docker-compose"

- name: Install self-hosted
uses: nick-fields/retry@v3
with:
max_attempts: 3
timeout_minutes: 10
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any chance this could actually increase the flake rate, due to timeouts becoming more frequent? Or are flakes caused by the actions itself timing out?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The typical install logic takes around 4-5 minutes. I doubled that time for the timeout so I don't think this should ever increase the flake rate.

retry_on: error
command: |
export REPORT_SELF_HOSTED_ISSUES=0
./install.sh

- name: Integration Test
run: ./integration-test.sh
uses: nick-fields/retry@v3
with:
max_attempts: 3
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just so I understand: what was the behavior before you added these settings? Unlimited retries? No timeout so it hung until the action crashed?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The tests would fail from flakes way too often, adding this in drastically increases the chance the tests pass when they should

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, in effect, the previous (implicit) setting was max_attempts: 1? Or timeout_minutes: Infinity? Or some combination of the two? I get that the problem we are trying to solve is flakiness, I'm just not clear how changing (raising? lowering?) max_attempts and timeout_minutes helps that, since it's not obvious what the current state of affairs is.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, that's correct. The max_attempts really is just the first step into adding flaky test detection. If a job fails, but then is retried and succeed, it can be marked as flaky. The timeout_minutes is a required parameter here. I can remove this and readd in a follow-up PR if that is more clear

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, that's fine. I just wanted to understand the change. LGTM!

timeout_minutes: 25
retry_on: error
command: ./integration-test.sh --${{ matrix.test-group }}

- name: Inspect failure
if: failure()
Expand Down
11 changes: 8 additions & 3 deletions _integration-test/run.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,12 @@ teardown() {
DID_TEAR_DOWN=1

if [ "$1" != "EXIT" ]; then
echo "An error occurred, caught SIG$1 on line $2"
set +x
error_msg="An error occurred, caught SIG$1 on line $2"
echo "$error_msg"
dsn="https://[email protected]/6627632"
local sentry_cli="docker run --rm -v $(pwd):/work -e SENTRY_DSN=$dsn getsentry/sentry-cli"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Odd to me that we set REPORT_ISSUES=0 above, but then send an envelope here anyway? Maybe I am misunderstanding something, but why would we not just do REPORT_ISSUES=1, and maybe figure out some way to configure it to send to an "integration tests" or similar project on our actual dogfood instance?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, I can incorporate this into the existing logic for REPORT_ISSUES I believe.

$sentry_cli send-event -m "$error_msg" --logfile $log_file
fi

echo "Tearing down ..."
Expand All @@ -41,10 +46,10 @@ echo "${_endgroup}"
echo "${_group}Starting Sentry for tests ..."
# Disable beacon for e2e tests
echo 'SENTRY_BEACON=False' >>$SENTRY_CONFIG_PY
echo y | $dcr web createuser --force-update --superuser --email $TEST_USER --password $TEST_PASS
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

docker run is slower than just an exec into a running container

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is but it is also intentional to keep the run separate from the main web process.

That said for a one-off thing like this, I think using exec is a good compromise if it saves a notable amount of time. I'd just want it documented with a brief comment.

$dc up -d
printf "Waiting for Sentry to be up"
timeout 90 bash -c 'until $(curl -Isf -o /dev/null $SENTRY_TEST_HOST); do printf '.'; sleep 0.5; done'
echo y | $dc exec web sentry createuser --force-update --superuser --email $TEST_USER --password $TEST_PASS
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happens if we exec into the web container but it isn't ready yet? Does docker wait until it is ready, or does this fail? If it's the former, we should echo "Waiting for Sentry..." before this runs, otherwise the user may be waiting a while. If it fails, we should add some sort of sync point to wait for the container to be up before trying this.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Docker compose up will only succeed if the container healthcheck for web passes, so I don't think this will be a problem. That is performed on a previous line, so when the tests get to the createuser logic the web container will always be ready

printf "Waiting for Sentry to be up"
echo ""
echo "${_endgroup}"

Expand Down
34 changes: 20 additions & 14 deletions integration-test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,24 +6,30 @@ rm -f sentry/enhance-image.sh
rm -f sentry/requirements.txt
export REPORT_SELF_HOSTED_ISSUES=0

echo "Testing initial install"
./install.sh
_integration-test/run.sh
_integration-test/ensure-customizations-not-present.sh
_integration-test/ensure-backup-restore-works.sh
test_option="$1"

echo "Make customizations"
cat <<EOT >sentry/enhance-image.sh
if [[ "$test_option" == "--initial-install" ]]; then
echo "Testing initial install"
_integration-test/run.sh
_integration-test/ensure-customizations-not-present.sh
_integration-test/ensure-backup-restore-works.sh
elif [[ "$test_option" == "--customizations" ]]; then
echo "Testing customizations"
source install/dc-detect-version.sh
$dc up -d
echo "Making customizations"
cat <<EOT >sentry/enhance-image.sh
#!/bin/bash
touch /created-by-enhance-image
apt-get update
apt-get install -y gcc libsasl2-dev python-dev libldap2-dev libssl-dev
EOT
chmod +x sentry/enhance-image.sh
printf "python-ldap" >sentry/requirements.txt
chmod +x sentry/enhance-image.sh
printf "python-ldap" >sentry/requirements.txt

echo "Testing in-place upgrade and customizations"
./install.sh --minimize-downtime
_integration-test/run.sh
_integration-test/ensure-customizations-work.sh
_integration-test/ensure-backup-restore-works.sh
echo "Testing in-place upgrade and customizations"
./install.sh --minimize-downtime
_integration-test/run.sh
_integration-test/ensure-customizations-work.sh
_integration-test/ensure-backup-restore-works.sh
fi
Loading