Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add new checksum refreshing infrastructure #38465

Merged
merged 3 commits into from
Nov 17, 2020
Merged

Conversation

staticfloat
Copy link
Sponsor Member

This changes the contrib/refresh_bb_tarballs.sh into a
contrib/refresh_checksums.mk which can be run in parallel much
faster than the original script thanks to the new checksum-foo targets
for all checksummable targets.

This will allow us to build a CI bot that automatically does this for
all PRs, hopefully noticing cases where the checksums are
missing/extraneous ones have not been pruned.

@staticfloat
Copy link
Sponsor Member Author

The proposed CI bot will do, essentially,

rm -rf deps/checksums
make -f contrib/refresh_checksums.mk -j${NPROC}
[ -z $(git status --porcelain) ]

The happy path is that the checksums all get recreated instantly. However, if some checksums fail to be recreated, this means the user should have pruned those checksums, or if there are new checksums, the user should have added those checksums. In this manner, I hope to avoid Fredrik having to post another Bernie Sanders meme at me. ;)

This changes the `contrib/refresh_bb_tarballs.sh` into a
`contrib/refresh_checksums.mk` which can be run in parallel _much_
faster than the original script thanks to the new `checksum-foo` targets
for all checksummable targets.

This will allow us to build a CI bot that automatically does this for
all PRs, hopefully noticing cases where the checksums are
missing/extraneous ones have not been pruned.
deps/blas.mk Outdated Show resolved Hide resolved
Copy link
Sponsor Member

@vtjnash vtjnash left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wouldn't this be easier as a direct bash script, without the Makefile generator, since you haven't actually defined any real targets? 😂

@StefanKarpinski
Copy link
Sponsor Member

I've been thinking for a while that we should take a lot of the intense functionality that's crammed into our makefiles and turn them into plain old scripts instead and simplify the makefiles. Of course, I also think we should stop supporting in-tree builds of our deps and only support BB or system builds with a separate repo for building our own copies of all the deps from source.

@staticfloat
Copy link
Sponsor Member Author

Wouldn't this be easier as a direct bash script, without the Makefile generator, since you haven't actually defined any real targets? 😂

Ironically, I started with a direct bash script, but the parallelism here is so important that I ended up just using make so that I can easily -j this. I was halfway to building my own semaphore/FIFO-based job queuing system based on this SO when I stopped myself and just used make.

Co-authored-by: Jameson Nash <[email protected]>
@staticfloat staticfloat merged commit 85000b1 into master Nov 17, 2020
@staticfloat staticfloat deleted the sf/checksum_infra branch November 17, 2020 22:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants