expand pip install

[Borda]

Description

this allows pip to resolve eventual version conflicts within particular layers
as it was now, there was no guarantee that the next installed package would break the previous installation (for example, min NumPy version)

Follow-up

I would suggest also doing it in the remaining dockers.
Moreover, reducing duplication may simplify the readability, aka define required packages in extra files and install them within the docker build...
(if agreed, I am happy to make both suggestions 🐰 )

[Borda]

is there a way how to check why the build is failing?
maybe #1077 could help here 🐰

[rosbo]

I'm currently moving our build to a templatized build and will incorporate this idea after I merge #1078

[Borda]

I'm currently moving our build to a templatized build and will incorporate this idea after I merge #1078

@rosbo I a not sure why it would not be compatible, the idea is to decouple the packages/version from the Docker structure so it will be lighter and easier to compare differences...

[Borda]

@rosbo is there an interest in finishing this one, if yes, I am happy to resolve conflicts and continue... 🐰

[rosbo]

Looping in @djherbis who is now responsible to manage this docker image / repo.

[djherbis]

@Borda I think this is a good change, though I think the main branch has drifted a decent amount from where this is.
If you can resolve the conflicts, feel free to ping me and I'll review.
I think this is a good step, I think the only concern I might have is if it might slow installation time if it has to do complex version matching but happy to try.

[Borda]

If you can resolve the conflicts, feel free to ping me and I'll review.

yes, happy to do it (that was my concern if there is interest in this direction so the work wont land flat 🐿️ )

[Borda]

@djherbis just cleaned out the PR - reset to head and add all changes from the ground, so it shall be fine now... just curious is there a CI to verify these changes? as discussed in #1077 and #1078 🐿️

[djherbis]

@Borda There is a CI, however it's private so only Kaggle sees the output.
I'll let you know when/if CI passes.

We should eventually try to get the build to work on a public CI as well.

[Borda]

I'll let you know when/if CI passes.

cool, thanks 🐰

[djherbis]

Thanks for all your hard work @Borda everything passed :) great work!