Support conda envs when using Miniconda

[pytoxbot]

• Bitbucket: https://bitbucket.org/hpk42/tox/issue/273
• Originally reported by: mnazbro
• Originally created at: 2015-09-11T20:57:05.200

Miniconda provides conda envs to create isolated Python environments similar to virtualenv. Its main advantage is that conda environments also isolate the binary shared library dependencies from the rest of your machine. Unfortunately, conda does not support virtualenvs because of the difference in how they manage shared libraries.

Since Miniconda isolates its shared library dependencies from the rest of your machine, virtualenvs which are created by a Miniconda-based environment fail during creation time. The virtualenvs will attempt to use shared libraries in the system-level location, but not the Miniconda location.

NOTE: If you try to create virtualenvs using Miniconda in an environment which already has the correct shared libraries in the system-level location, then your virtualenv will seem to succeed. However, any shared libraries which were only installed using conda will fail at runtime. Because of this, testing this feature should be done in an isolated environment.

I believe that the first implementation of this feature has these requirements:

• Create conda environments instead of virtualenvs based on a single miniconda Python version
• Install pip using conda by default
• Add dependencies using pip (like virtualenvs)

Further features which are necessary for complete conda env supports are:

• Create conda environments for multiple miniconda Python versions
• Allow the installation of conda packages along with pip packages

There is a similar project called ctox (https://github.com/hayd/ctox) which tries to implement this. However, it doesn't have any guarantees on reliability. This project was actually started around the time there were comments on this ticket (https://bitbucket.org/hpk42/tox/issues/167/add-an-option-to-run-commands-after) about conda.

I would like to work with the tox team on specifying any additions necessary for conda support in Tox.

[pytoxbot]

Original comment by @nicoddemus

Hey @kalefranz,

Thanks for chiming in. I was also involved in the discussion which lead to the extend-envs-and-packagebuilds.md document. That draft was the result of brainstorming and looking at the current implementation to figure out what was feasible/doable, so it should be a good starting point if someone wants to champion it; both the section about how each plugin can handle different packages as well as the hooks needed for the plugin system to be generic were written with the thought of supporting both virtualenv and conda as plugins (with virtualenv being a core plugin).

Unfortunately I'm also short on time to work on this right now although I would love to.

[pytoxbot]

Original comment by kalefranz

I'm the lead for the conda project, and I just want to chime in to say I'm extremely supportive of conda and tox working together. Conda envs and python virtualenvs don't currently "play nice" together, but there's no technical barrier to why they couldn't/shouldn't. If they did, tox would work just fine for python projects within a conda env. However, making a conda plugin for tox is a more powerful concept: it would allow tox to create environment matrices for any language ecosystem.

I don't currently have the bandwidth to do this work myself, which is why in other places I've asked for community help. However, if there are flags or features that conda is missing to make everything work, I'm eager to help make it happen.

Related:

• https://groups.google.com/a/continuum.io/forum/#!topic/conda/63B0jnPR-V4
• Patch and add virtualenv conda-forge/staged-recipes#1139

[pytoxbot]

Original comment by @obestwalter

The draft mentioned in #338 is the outcome of us sitting together at the sprint and sketching out how this should look like from the user perspective and make sure it would be general enough to make it work with other scenarios (like no virtualenv or pyenv, etc.). If someone is interested in starting to work on the implementation this can act as the starting point. I am not using conda myself, so I am not a good candidate to implement the conda specific part.

[pytoxbot]

Original comment by Peque

@obestwalter Please, let us know if you have something testable (I would be glad to try it out). 😊

[pytoxbot]

Original comment by @obestwalter

We started working out a concept to make things like this possible. It is also very likely that conda will be proof of concept to show the viabilty of this approach. See #338

[pytoxbot]

Original comment by @ssbarnea

@panicking any updates regarding the possible marriage between tox and conda?

[pytoxbot]

Original comment by mnazbro

In December I made a prototype to get this to work with only Conda (and not virtualenv). I haven't quite gotten the chance to make it play nicely with all of the tests. I'll go ahead and get back to working on making it fit into Tox better.

[pytoxbot]

Original comment by @flub

See also the discussion at https://bitbucket.org/hpk42/tox/issues/313/handle-mutliple-backends-for-virtual

I wonder if these issues should be merged or one of them marked as duplicate?

[pytoxbot]

Original comment by @hpk42

Sorry it took a while. Me and i think @flub would also like to think further on supporting non-virtualenv virtualization through hooks and a tox-conda or tox-miniconda plugin. If you'd like to help make this happen you are welcome to discuss, and also to join and/or contribute to https://www.indiegogo.com/projects/python-testing-sprint-mid-2016/x/4034848#/ -- i'd be happy if some more people, companies fund the sprint with a particular focus on tox which needs more activity and community.

[cjermain]

What is the current status of conda support? It sounds like from this thread that there may have been a working draft at some point.

[obestwalter]

Hi @cjermain the drafts are here: https://github.com/tox-dev/tox/tree/2.7.0/doc/drafts

the idea is to extend the tox hooks in a way that conda integration can be implemented and other forms of virtualization also.

There is no one yet though who wants this enough to actuallly do the work.

[ramanshah]

@obestwalter From my vantage point in the data science world, the need for tox to wrangle conda environments remains acute :) The problem is that pip/virtualenv is a non-starter for data science products that are to run on a variety of platforms (e.g., Linux): there are ubiquitous dependencies in the quantitative stack comprised of old Fortran code, and often one wants/needs optimized versions of these binary dependencies.

It's bad enough that I'm seeing a whole crop of data scientists today who assume that rolling a junky shell script is the best they'll be able to do in organizing their testing process.

I would be thrilled if I could help. It would be an extracurricular for me, as I assume it is for most of us. I've been making Python do math and science for 15 years, but I'm regrettably ignorant of Python build system internals, etc. So I have a lot to learn.

How should I start? I'm happy to tackle a small PR paying some technical debt if it would educate me on the codebase and lay a brick toward this sorely needed feature.

[gaborbernat]

@ramanshah I would suggest taking a look at the other issues and diving into one you feel comfortable with

[ramanshah]

Of course, thanks. In the process, I appreciate any insider hints about which tasks are both relevant and tractable for a beginner to the project.

[obestwalter]

Hi @ramanshah thanks for jumping in. It's great that you want to improve testing practices in the data science community ❤️

The basic idea of this change (anyone correct me if I am wrong - it's a while that I had a look at the topic) is to be able to extend tox more for other venv handlers and package managers by moving some of the things that are now hard wired into hooks that are then used by tox core for its default functionality and make switching of behaviour simpler - either by writing a plugin or - in the case of adding conda support using different hooks depending on what is needed.

There is also an abandoned looking project by @hayd I never looked closely but that might give you some ideas: https://github.com/hayd/ctox

If you hit a wall or don't understand the concept of hooks yet, have a look at existing plugins in the tox-dev org. It is definitely solvable without using the hook system, but my feeling is that we should use them more extensively also for internal stuff.

[ramanshah]

Wonderful, @obestwalter - that helps a lot already. I'm eager to scrape together some hobby-coding time and get some momentum going.

[DancingQuanta]

I am keen on this too, especially in data science.

[obestwalter]

I self assigned this now to create a tox-conda plugin.

Reason: I am attending Pythoncamp in Cologne this weekend. There are usually some folks who know their way around this conda stuff. My main problem is that I don't use it myself. I will propose a session where I offer my tox knowledge in exchange for the necessary conda knowledge to create a tox-conda plugin based on the next tox version (if I get my PR merged until then).

If I kickstart this it would be great if then one or more actual conda users step up to become maintainers for the plugin ❤️.

[DancingQuanta]

Conferences are a great way to attract attention to your work. Good luck!

[ramanshah]

So excited for this! I'm sorrowful that I haven't been able to get momentum on this. Learning a new codebase in free time is such an uphill battle...

[nicoddemus]

I remember @flub and I got stuck at this during the 2016 sprint because conda is both a package creation tool (similar to python setup.py sdist) and environment manager (similar to pip), which complicates things if we want tox to use conda to generate a package and test it.

But if we want to change tox to support only environment creation but keep creating packages using python setup.py sdist (or even allowing users to test directly from source by using skip_sdist option) then I believe things are much simpler.

Just wanted to mention that I think pursuing the latter is very worthwhile and will solve most use cases from conda users that want to use tox. 😁

[obestwalter]

Hey @nicoddemus - thanks for your input. I will keep that in mind and have a proper look at the drafts we created in 2016. I hope some seasoned conda users will be in Cologne and we get at least a minimal implementation done that will make 80% of the people happy in 20% of the time. Wish me luck :)

Hi @ramanshah, no worries. Maybe you can join in once the ground work is layed out and you get an idea how stuff plays together in the context of tox.

[kalefranz]

If anyone is at PyCON and interested in making progress on this, I'm happy to chat and help iron out any issues/discrepancies that might come up with using conda instead of pip + virtualenv. I'll be around through the full sprint week.

[gaborbernat]

@kalefranz I'll be happy to chip in on this. If nothing else let's have a chat of what this would require.

[obestwalter]

Great! If some actual conda users would stick their heads together we might finally get somewhere with this. https://github.com/tox-dev/tox-conda is waiting to be filled :)

Have fun at PyCon!

[obestwalter]

I just fixed some settings and links in https://github.com/tox-dev/tox-conda and activated travis and appveyor CI, so you're good to go.

It is likely that you might need new hooks to implement this properly, so I expect you will need to open a PR here then, so that we can get that added as well.

cc @nicoddemus - it would be great if you could also spare some time to have a look - hope you have recovered from that nasty flu 🤗

[gaborbernat]

Done via https://github.com/tox-dev/tox-conda

GitHub

tox-dev/tox-conda

Make tox cooperate with conda envs. Contribute to tox-dev/tox-conda development by creating an account on GitHub.

[ramanshah]

I'm really excited to try this out!!!