Output extraction performance tweaks #45
Conversation
This gives an performance improvement from 175 to 144 seconds on tmrow.
|
Nice, I've triggered a bunch of builds for testing the impact of this:
Will give an update tomorrow on how it went :) |
|
Don't seem like it changed much, will investigate if it was actually working as intended :) |
Sad. What is the usual running time for a CI job where nothing changed? It is likely that I hope that #46 will show a serious performance increase. That PR also includes this branch, so you can test both if you want to. Note that #46 will only work after the build has run once (as images needs to be updated with the dependency hash as part of their Dockerfile). |
Approximately 8 minutes. So definitely room for improvements! 😅
I guess that could be, there's not a huge improvements at least - but it's kind of hard to actually see changes as the speeds differ quite a lot. Below is a sample of two builds with
I'll test with #46 :) |
There was a problem hiding this comment.
I think there's still room for improving the output extraction - for one of our packages (bloom) it still takes 4s on a noop run, but this seems like a good first step and then we can more easily measure improvements of it going forward (due to #46 removing all the "noise"). So LGTM! 💪
(Pulled out from #44). Fixes #42. Closes #2
The output extraction done during
brick buildis rather slow. I noticed a comment in the code to useget_archiveinstead of mounting volumes, and doing so improves the performance a bit.A
brick -r buildon tmrow on my local machine when all images are already build: from 198-228 to 156-164 seconds (two measurements).I also did some other minor clean up while poking around and preparing for cache optimizations.
Future steps
If we simply remove all output extraction we get down to 122 seconds (compared to around ~210 seconds from above). This shows that there is still a lot to gain from optimizing the extraction.