Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: failing gh actions due to runner out of disk space #7565

Merged
merged 1 commit into from
Aug 5, 2024

Conversation

kevin9foong
Copy link
Member

@kevin9foong kevin9foong commented Aug 5, 2024

Issue:

backend_test cache node modules step failing due to out of memory: cache lookup restores the tar file in memory, but when unzipped to storage, out of disk space memory occurs. This causes all our github actions to fail.
The error message is the github runnner running out of disk storage space when extracting the restored cached tar file.

/usr/bin/tar -z -xf /home/runner/work/_temp/20c9e7c9-42f5-4d22-8c2c-7e6e58b541ec/cache.tgz -P -C /home/runner/work/FormSG/FormSG
[14](https://github.com/opengovsg/FormSG/actions/runs/10218808254/job/28333961131#step:6:15)
/usr/bin/tar: node_modules/@datadog/native-appsec/prebuilds/linux-x64/libddwaf.so: Wrote only 8704 of 10240 bytes
[15](https://github.com/opengovsg/FormSG/actions/runs/10218808254/job/28333961131#step:6:16)
/usr/bin/tar: node_modules/@datadog/native-appsec/prebuilds/linux-arm64: Cannot mkdir: No space left on device
[16](https://github.com/opengovsg/FormSG/actions/runs/10218808254/job/28333961131#step:6:17)
/usr/bin/tar: node_modules/@datadog/native-appsec/prebuilds/linux-arm64: Cannot mkdir: No space left on device
[17](https://github.com/opengovsg/FormSG/actions/runs/10218808254/job/28333961131#step:6:18)
/usr/bin/tar: node_modules/@datadog/native-appsec/prebuilds/linux-arm64/node-napi.node: Cannot open: No such file or directory
[18](https://github.com/opengovsg/FormSG/actions/runs/10218808254/job/28333961131#step:6:19)
/usr/bin/tar: node_modules/@datadog/native-appsec/prebuilds/linux-arm64: Cannot mkdir: No space left on device

Fix:

Remove the code to generate swapfile since it is no longer needed.

Explanation:

running df -h shows we have 19GB of disk space (despite 14GB indicated by github). We previously allocate 15GB to swapfile, leaving 4GB for storage.
Recently, the disk space usage has likely exceeded the 4GB and hence caused this OO disk space error.

Since GH runners now provide 16GB of memory instead of the previous 7GB, we do not need to manually add this swapfile any more.

Additional context:

From @g-tejas using wayback machine:
image
back when swap file of 15G was selected, there was only 7GB of RAM allocated to standard github runners. Today, there is 16GB allocated to standard github runners. Hence, we do not need the extra RAM for github runner.

@kevin9foong kevin9foong requested review from g-tejas, sebastianwzq and KenLSM and removed request for sebastianwzq August 5, 2024 06:54
@kevin9foong kevin9foong changed the title fix: reduce swapfile memory size from 15G to 4G fix: reduce swapfile memory size for backend- Aug 5, 2024
@kevin9foong kevin9foong changed the title fix: reduce swapfile memory size for backend- fix: reduce swapfile memory size for backend-tests to free up disk space Aug 5, 2024
@kevin9foong kevin9foong changed the title fix: reduce swapfile memory size for backend-tests to free up disk space fix: failing gh actions by reducing swapfile memory size for backend-tests to free up disk space Aug 5, 2024
@kevin9foong kevin9foong force-pushed the fix/gh-actions-out-of-disk-space-failure branch from bca1795 to 2369089 Compare August 5, 2024 09:59
@kevin9foong kevin9foong changed the title fix: failing gh actions by reducing swapfile memory size for backend-tests to free up disk space fix: failing gh actions due to runner out of disk space Aug 5, 2024
@kevin9foong kevin9foong self-assigned this Aug 5, 2024
Copy link
Contributor

@KenLSM KenLSM left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Always happy to see that we're removing things.

@kevin9foong kevin9foong merged commit 0d060e0 into develop Aug 5, 2024
18 checks passed
@kevin9foong kevin9foong deleted the fix/gh-actions-out-of-disk-space-failure branch August 5, 2024 10:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants