-
-
Notifications
You must be signed in to change notification settings - Fork 509
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support conda envs when using Miniconda #273
Comments
Original comment by @nicoddemus Hey @kalefranz, Thanks for chiming in. I was also involved in the discussion which lead to the extend-envs-and-packagebuilds.md document. That draft was the result of brainstorming and looking at the current implementation to figure out what was feasible/doable, so it should be a good starting point if someone wants to champion it; both the section about how each plugin can handle different packages as well as the hooks needed for the plugin system to be generic were written with the thought of supporting both virtualenv and conda as plugins (with virtualenv being a core plugin). Unfortunately I'm also short on time to work on this right now although I would love to. |
Original comment by kalefranz I'm the lead for the conda project, and I just want to chime in to say I'm extremely supportive of conda and tox working together. Conda envs and python virtualenvs don't currently "play nice" together, but there's no technical barrier to why they couldn't/shouldn't. If they did, tox would work just fine for python projects within a conda env. However, making a conda plugin for tox is a more powerful concept: it would allow tox to create environment matrices for any language ecosystem. I don't currently have the bandwidth to do this work myself, which is why in other places I've asked for community help. However, if there are flags or features that conda is missing to make everything work, I'm eager to help make it happen. Related: |
Original comment by @obestwalter The draft mentioned in #338 is the outcome of us sitting together at the sprint and sketching out how this should look like from the user perspective and make sure it would be general enough to make it work with other scenarios (like no virtualenv or pyenv, etc.). If someone is interested in starting to work on the implementation this can act as the starting point. I am not using conda myself, so I am not a good candidate to implement the conda specific part. |
Original comment by Peque @obestwalter Please, let us know if you have something testable (I would be glad to try it out). 😊 |
Original comment by @obestwalter We started working out a concept to make things like this possible. It is also very likely that conda will be proof of concept to show the viabilty of this approach. See #338 |
Original comment by @ssbarnea @panicking any updates regarding the possible marriage between tox and conda? |
Original comment by mnazbro In December I made a prototype to get this to work with only Conda (and not virtualenv). I haven't quite gotten the chance to make it play nicely with all of the tests. I'll go ahead and get back to working on making it fit into Tox better. |
Original comment by @flub See also the discussion at https://bitbucket.org/hpk42/tox/issues/313/handle-mutliple-backends-for-virtual I wonder if these issues should be merged or one of them marked as duplicate? |
Original comment by @hpk42 Sorry it took a while. Me and i think @flub would also like to think further on supporting non-virtualenv virtualization through hooks and a tox-conda or tox-miniconda plugin. If you'd like to help make this happen you are welcome to discuss, and also to join and/or contribute to https://www.indiegogo.com/projects/python-testing-sprint-mid-2016/x/4034848#/ -- i'd be happy if some more people, companies fund the sprint with a particular focus on tox which needs more activity and community. |
What is the current status of conda support? It sounds like from this thread that there may have been a working draft at some point. |
Hi @cjermain the drafts are here: https://github.com/tox-dev/tox/tree/2.7.0/doc/drafts the idea is to extend the tox hooks in a way that conda integration can be implemented and other forms of virtualization also. There is no one yet though who wants this enough to actuallly do the work. |
@obestwalter From my vantage point in the data science world, the need for tox to wrangle conda environments remains acute :) The problem is that pip/virtualenv is a non-starter for data science products that are to run on a variety of platforms (e.g., Linux): there are ubiquitous dependencies in the quantitative stack comprised of old Fortran code, and often one wants/needs optimized versions of these binary dependencies. It's bad enough that I'm seeing a whole crop of data scientists today who assume that rolling a junky shell script is the best they'll be able to do in organizing their testing process. I would be thrilled if I could help. It would be an extracurricular for me, as I assume it is for most of us. I've been making Python do math and science for 15 years, but I'm regrettably ignorant of Python build system internals, etc. So I have a lot to learn. How should I start? I'm happy to tackle a small PR paying some technical debt if it would educate me on the codebase and lay a brick toward this sorely needed feature. |
@ramanshah I would suggest taking a look at the other issues and diving into one you feel comfortable with |
Of course, thanks. In the process, I appreciate any insider hints about which tasks are both relevant and tractable for a beginner to the project. |
Hi @ramanshah thanks for jumping in. It's great that you want to improve testing practices in the data science community ❤️ The basic idea of this change (anyone correct me if I am wrong - it's a while that I had a look at the topic) is to be able to extend tox more for other venv handlers and package managers by moving some of the things that are now hard wired into hooks that are then used by tox core for its default functionality and make switching of behaviour simpler - either by writing a plugin or - in the case of adding conda support using different hooks depending on what is needed. There is also an abandoned looking project by @hayd I never looked closely but that might give you some ideas: https://github.com/hayd/ctox If you hit a wall or don't understand the concept of hooks yet, have a look at existing plugins in the tox-dev org. It is definitely solvable without using the hook system, but my feeling is that we should use them more extensively also for internal stuff. |
Wonderful, @obestwalter - that helps a lot already. I'm eager to scrape together some hobby-coding time and get some momentum going. |
I am keen on this too, especially in data science. |
I self assigned this now to create a Reason: I am attending Pythoncamp in Cologne this weekend. There are usually some folks who know their way around this conda stuff. My main problem is that I don't use it myself. I will propose a session where I offer my tox knowledge in exchange for the necessary conda knowledge to create a tox-conda plugin based on the next tox version (if I get my PR merged until then). If I kickstart this it would be great if then one or more actual conda users step up to become maintainers for the plugin ❤️. |
Conferences are a great way to attract attention to your work. Good luck! |
So excited for this! I'm sorrowful that I haven't been able to get momentum on this. Learning a new codebase in free time is such an uphill battle... |
I remember @flub and I got stuck at this during the 2016 sprint because But if we want to change Just wanted to mention that I think pursuing the latter is very worthwhile and will solve most use cases from |
Hey @nicoddemus - thanks for your input. I will keep that in mind and have a proper look at the drafts we created in 2016. I hope some seasoned conda users will be in Cologne and we get at least a minimal implementation done that will make 80% of the people happy in 20% of the time. Wish me luck :) Hi @ramanshah, no worries. Maybe you can join in once the ground work is layed out and you get an idea how stuff plays together in the context of tox. |
If anyone is at PyCON and interested in making progress on this, I'm happy to chat and help iron out any issues/discrepancies that might come up with using conda instead of pip + virtualenv. I'll be around through the full sprint week. |
@kalefranz I'll be happy to chip in on this. If nothing else let's have a chat of what this would require. |
Great! If some actual conda users would stick their heads together we might finally get somewhere with this. https://github.com/tox-dev/tox-conda is waiting to be filled :) Have fun at PyCon! |
I just fixed some settings and links in https://github.com/tox-dev/tox-conda and activated travis and appveyor CI, so you're good to go. It is likely that you might need new hooks to implement this properly, so I expect you will need to open a PR here then, so that we can get that added as well. cc @nicoddemus - it would be great if you could also spare some time to have a look - hope you have recovered from that nasty flu 🤗 |
Done via https://github.com/tox-dev/tox-conda
|
I'm really excited to try this out!!! |
Miniconda provides conda envs to create isolated Python environments similar to
virtualenv
. Its main advantage is that conda environments also isolate the binary shared library dependencies from the rest of your machine. Unfortunately, conda does not support virtualenvs because of the difference in how they manage shared libraries.Since Miniconda isolates its shared library dependencies from the rest of your machine, virtualenvs which are created by a Miniconda-based environment fail during creation time. The virtualenvs will attempt to use shared libraries in the system-level location, but not the Miniconda location.
NOTE: If you try to create virtualenvs using Miniconda in an environment which already has the correct shared libraries in the system-level location, then your virtualenv will seem to succeed. However, any shared libraries which were only installed using conda will fail at runtime. Because of this, testing this feature should be done in an isolated environment.
I believe that the first implementation of this feature has these requirements:
Further features which are necessary for complete conda env supports are:
There is a similar project called ctox (https://github.com/hayd/ctox) which tries to implement this. However, it doesn't have any guarantees on reliability. This project was actually started around the time there were comments on this ticket (https://bitbucket.org/hpk42/tox/issues/167/add-an-option-to-run-commands-after) about conda.
I would like to work with the tox team on specifying any additions necessary for conda support in Tox.
The text was updated successfully, but these errors were encountered: