Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow to skip the save post-step #489

Closed
wants to merge 8 commits into from

Conversation

roni-frantchi
Copy link

@roni-frantchi roni-frantchi commented Dec 14, 2020

This is similar to #474 - but improves on that by skipping the post (save) step entirely, saving tens of seconds on each run.

Fixes #350
Fixes #334

@roni-frantchi roni-frantchi requested a review from a team as a code owner December 14, 2020 09:07
@domjtalbot
Copy link

I really like this approach! It's helped to save more valuable seconds on my builds than other solutions.
What do you think about the variable name being something like READ_ONLY?

@roni-frantchi
Copy link
Author

Thanks @domjtalbot.
Sure thing.
If we get a positive response from maintainers on pushing it in, I'd be happy to make that adjustment along with any others suggested.

Copy link
Member

@joshmgross joshmgross left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your suggestion!

Due to the type issues with env always being a string, I think we'd need this to be an input rather than an environment variable so that we can properly evaluate it.

Also this does not make the cache key option, so could you update your PR description to remove "Fixes #210"?

README.md Outdated Show resolved Hide resolved
action.yml Outdated Show resolved Hide resolved
Copy link
Author

@roni-frantchi roni-frantchi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for reviewing @joshmgross !

I think we'd need this to be an input rather than an environment variable so that we can properly evaluate it

post-if is undocumented and despite my best efforts to make it evaluate inputs I couldn't - only env was available to me.

Due to the type issues with env always being a string...

I did address this one though - so perhaps using env would be acceptable now?

action.yml Outdated Show resolved Hide resolved
README.md Outdated Show resolved Hide resolved
@anders-kiaer
Copy link

I see this PR has gone through a first review by @joshmgross already. Will it be merged in the near future, or should use cases needing to skip the save step use e.g. a fork of this action?

Regarding using environment variable: I think this is a good choice, if my understanding is correct that previous steps in the job then can dynamically/during runtime instruct the cache action save/post step through https://docs.github.com/en/actions/reference/workflow-commands-for-github-actions#setting-an-environment-variable.

Thanks 🙂

@mikeage
Copy link

mikeage commented Apr 5, 2021

This looks wonderful; one question, if you don't mind. I have a very large cache (~750 MB), and when I have multiple PRs in progress, it appears that the 5GB cache gets filled up with those, and the cache from main gets pushed out. If I were to use this fork and set the option to skip the saving on the PRs but not on the main branch, this would keep the main cache as a base for the all of the PRs, right?

@zwass
Copy link

zwass commented Apr 16, 2021

@joshmgross Can you please re-review/merge this? I'd like to use it and would prefer not to rely on a fork.

@joshmgross joshmgross requested a review from a team April 16, 2021 20:15
@joshmgross
Copy link
Member

👋 Hey @zwass, the @actions/artifacts-actions team should be able to help out here.

Copy link
Member

@dhadka dhadka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TY for proposing this change. Just some minor comments on the README changes, but otherwise looks good. Please ping me when ready and I can 🚢 this.

README.md Outdated Show resolved Hide resolved
@@ -67,6 +70,8 @@ jobs:
- uses: actions/checkout@v2

- name: Cache Primes
env:
CACHE_SKIP_SAVE: true
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given that this is the default example and skipping cache save is a special case, I wouldn't include this here. Perhaps move this into the Examples.md file and we can add some specific examples of where this is useful.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Something we can also highlight in the example is how users can either set this on the cache step as shown here or using echo "CACHE_SKIP_SAVE=true" >> $GITHUB_ENV.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the changes you've suggested on the readme file capture it well enough

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@roni-frantchi & @dhadka, would be great if this PR could be merged to reduce the workflow runtime when a cache-update isn't required. Seems like only a small adjustment would be needed?
Not sure if this issue could be addressed in this PR too: skipping the post-step altogether when there was a cache-hit.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if this issue could be addressed in this PR too

I'm not sure that it can either...
Seeing so many are expecting it, maybe we should explore doing it on another PR?..

Seems like only a small adjustment would be needed?

Sorry @rene-leanix it's been a while - what adjustment would that be?..

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@roni-frantchi, I was just referring to @dhadka's comments above in this thread: here and here.

README.md Outdated
@@ -78,6 +83,8 @@ jobs:
run: /generate-primes.sh -d prime-numbers

- name: Use Prime Numbers
env:
CACHE_SKIP_SAVE: true
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be removed now that it's set on the cache-primes step.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

README.md Outdated Show resolved Hide resolved
@roni-frantchi roni-frantchi requested review from dhadka and removed request for a team April 20, 2021 10:54
@ollydev
Copy link

ollydev commented Apr 20, 2021

@dhadka This is what I wanted #498 for, didn't see this PR though.

Right now cache will never be saved if the job fails.
Example: Spend 10mins building dependencies (succesfully) but a future job (such as failing a test) fails. Cache will not save!

Can we add something like:

env:  
  CACHE_FORCE_SAVE: true

I think it would be something like this:

post-if: success() && env.CACHE_SKIP_SAVE != 'true' || failed() && env.CACHE_FORCE_SAVE == 'true'

Thanks.

@zwass
Copy link

zwass commented Apr 20, 2021

Perhaps @ollydev's request could be merged into this one with a single flag?

CACHE_SAVE=always - Always save regardless of errors in preceding steps.
CACHE_SAVE=never - Never save regardless of errors.
CACHE_SAVE=default or Not set - Default behavior (as before this PR).

@jmoguillansky-gpsw
Copy link

jmoguillansky-gpsw commented Apr 29, 2021

failed() && env.CACHE_FORCE_SAVE == 'true': how do we know the content to cache was generated successfully? I think we need some mechanism to indicate that generating the cache content was successful, and then cache it immediately, why wait until the end of the workflow?

@vsvipul vsvipul assigned Phantsure and unassigned vsvipul Aug 9, 2022
@Phantsure Phantsure added area:granular-control Issues related to granular control in saving or restoring cache feature request labels Sep 7, 2022
armujahid added a commit to armujahid/deno that referenced this pull request Nov 4, 2022
Other two instances aren't updated. blocked by actions/cache#489 (or 571)
armujahid added a commit to armujahid/deno that referenced this pull request Nov 4, 2022
Other two instances aren't updated. blocked by actions/cache#489 (or 571)
@kotewar kotewar assigned tanuj077 and kotewar and unassigned Phantsure Nov 10, 2022
@iuioiua
Copy link

iuioiua commented Nov 18, 2022

@roni-frantchi, is it possible for you to update the branch? It'd remove the deprecation warnings around save-state, set-output and Node.js 12 for those using the fork.

@Jimimaku
Copy link

Jimimaku commented Nov 21, 2022

Dies ist ähnlich wie # 474 - verbessert dies jedoch, indem der (gespeicherte) Schritt vollständig übersprungen wird, wodurch bei jedem Lauf Dutzende von Sekunden gespart werden.post

Behebt #350 Behebt #334

@erik-krogh erik-krogh mentioned this pull request Nov 22, 2022
10 tasks
@kotewar
Copy link
Contributor

kotewar commented Dec 7, 2022

Hey all, 👋🏽

We have created a discussion with a proposal that will solve this problem, do let us know your feedback, thanks 😊

@kotewar
Copy link
Contributor

kotewar commented Dec 22, 2022

Hey everyone 👋🏽

Two new actions actions/cache/restore and actions/cache/save are now available with tag v3 to everyone for use. These can now be used to achieve granular control on the restore and save steps in the cache action.

Do try them and let us know your feedback. We hope these new actions will take care of your use cases. 🙇🏽

Closing this PR now as we believe the new actions will take care of the same, feel free to reopen it or create an issue if need be.

@kotewar kotewar closed this Dec 22, 2022
cammo1123 added a commit to cammo1123/serenity that referenced this pull request Mar 27, 2023
cammo1123 added a commit to cammo1123/serenity that referenced this pull request Mar 27, 2023
cammo1123 added a commit to cammo1123/serenity that referenced this pull request Mar 27, 2023
cammo1123 added a commit to cammo1123/serenity that referenced this pull request Mar 27, 2023
cammo1123 added a commit to cammo1123/serenity that referenced this pull request Mar 28, 2023
cammo1123 added a commit to cammo1123/serenity that referenced this pull request Mar 30, 2023
BertalanD added a commit to BertalanD/serenity that referenced this pull request May 27, 2023
This version now natively supports read-only caches (`cache/restore@v3`)
so we no longer need to pin the version to a commit in actions/cache#489
which is an unmerged PR.

The update is mostly mechanical:
- Steps with `CACHE_SKIP_SAVE` not set can use the plain `cache@v3`
  action.
- Steps with `CACHE_SKIP_SAVE` set to a constant `true` are changed to
  `cache/restore@v3`.
- Steps with saving disabled when running on a pull request are changed
  to a pair of `cache/restore@v3` and `cache/save@v3`. This setup is
  used for the large (100s of MB) ccache and Toolchain caches. As caches
  saved in pull requests can only be utilized from within the same PR,
  uploading these would only waste time and our storage quote.
  Therefore, we skip the `save` steps if running on a PR.

Co-authored-by: Cameron Youell <[email protected]>
ADKaster pushed a commit to SerenityOS/serenity that referenced this pull request May 28, 2023
This version now natively supports read-only caches (`cache/restore@v3`)
so we no longer need to pin the version to a commit in actions/cache#489
which is an unmerged PR.

The update is mostly mechanical:
- Steps with `CACHE_SKIP_SAVE` not set can use the plain `cache@v3`
  action.
- Steps with `CACHE_SKIP_SAVE` set to a constant `true` are changed to
  `cache/restore@v3`.
- Steps with saving disabled when running on a pull request are changed
  to a pair of `cache/restore@v3` and `cache/save@v3`. This setup is
  used for the large (100s of MB) ccache and Toolchain caches. As caches
  saved in pull requests can only be utilized from within the same PR,
  uploading these would only waste time and our storage quote.
  Therefore, we skip the `save` steps if running on a PR.

Co-authored-by: Cameron Youell <[email protected]>
JRedrupp pushed a commit to JRedrupp/serenity that referenced this pull request May 29, 2023
This version now natively supports read-only caches (`cache/restore@v3`)
so we no longer need to pin the version to a commit in actions/cache#489
which is an unmerged PR.

The update is mostly mechanical:
- Steps with `CACHE_SKIP_SAVE` not set can use the plain `cache@v3`
  action.
- Steps with `CACHE_SKIP_SAVE` set to a constant `true` are changed to
  `cache/restore@v3`.
- Steps with saving disabled when running on a pull request are changed
  to a pair of `cache/restore@v3` and `cache/save@v3`. This setup is
  used for the large (100s of MB) ccache and Toolchain caches. As caches
  saved in pull requests can only be utilized from within the same PR,
  uploading these would only waste time and our storage quote.
  Therefore, we skip the `save` steps if running on a PR.

Co-authored-by: Cameron Youell <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:granular-control Issues related to granular control in saving or restoring cache feature request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Read only cache? Support for Read Only Cache / Configurable Cache Saving