pip install extras requires and install requires, but not the package

[gaborbernat]

For library development it's often the case that you specify your test, publish, documentation requirements in extras; to have all dependencies in one place. There are many cases of this in wild, e.g. here.. In this case would be nice for development if we could install only extras and install requires libraries, but not the library itself. Would that be possible? E.g. with:

pip install ".[testing, docs]" --deps-only

Thanks,

[ghost]

For development, use pip install -e ..

[gaborbernat]

But what if you don't want to install the package itself? 🤔

[pfmoore]

I'm not sure this is a good use of extras, to be honest. Projects I've worked on typically specify development requirements in a requirements file, not as an extra. However, I find it hard to argue that @jaraco - the author of rwt, who is also the maintainer of setuptools, doesn't know how to use extras correctly! Maybe he can comment on how he would expect things to work in this case?

[gaborbernat]

The reason I don't like the requirements.txt approach (myself) is that then you have specified your dependencies with their version and environment markers in two places, and keeping them in sync can become a challenge.

[pfmoore]

But if you put your development dependencies in requirements.txt and just your runtime dependencies in setup.py where's the duplication?

Anyway, the point of the issue isn't to debate whether your approach is right or wrong - how you structure your project is your choice. The question here is whether wanting to install just the extras and not the actual project is something pip should support directly. From my reading of the setuptools documentation on extras, it doesn't sound like it's an intended use case for extras, and even if it were, it's a pretty unusual situation. So I'm probably -0 on the idea (I'm not actively against it, but I don't see the value being sufficient to justify the extra complexity in pip).

[gaborbernat]

your run-time dependencies are a subset of your development dependencies; there's the duplication 👍 Lately I've seen many-many projects using this way extras, hence my idea, should this be maybe be supported by pip. Another reason for extras is that binds well with having multiple tools for various tools (e.g. testing, documentation, linter) and each of them having their own separate venv with tox. But I think it's mainly having your dependencies in a single place.

[ghost]

But what if you don't want to install the package itself? 🤔

I think this is where you are confused. The package should be installed in editable mode for development (probably in a virtual environment) so that you can run the console and GUI scripts. And by
"installed," I mean that a 1KB .egg_info is placed in the site-packages. That is how pretty much every Python project works.

[jaraco]

For a while, I thought that requirements.txt files was the best place to declare dependencies for various purposes (tests, docs, development, etc). I was recommending storing test requirements in tests/requirements.txt and docs requirements in docs/requirements.txt.

Lately, I've been moving away from declaring various sets of dependencies in requirements files for a few reasons:

1. If using different files for different aspects, they'll inevitably end up referencing each other (or the package under test itself), but the relationship between these is not necessarily obvious or well-defined. Should requirements-dev also include tests/requirements and docs/requirements? Should it install . with -e or not?
2. While requirements.txt allows for additional -r directives to be given, the relationship between these is unclear if not unstable. That is, if docs/requirements.txt references -r ../requirements.txt, should that line be evaluated relative to the requirements.txt file or relative to the cwd of the invocation. I think the latter is what pip does, meaning the requirements.txt files are context-sensitive.
3. Some systems, like the Setuptools 'build_docs' commands, expect the command to be invoked from the root of the package, whereas systems like RTD expect commands to be invoked in the docs root. This forced disparity exacerbates (2).
4. In none of these cases are the dependencies available as package metadata. Using extras on the other hand exposes these requirements as package metadata (one can query for "what are the testing requirements for package X?" or ask to install X with its testing dependencies).
5. Tox has simple, elegant support for specifically including select extras. And although you can specify -rtests/requirements.txt in deps, it has errors if you try to put a space after -r. Plus, if those requirements.txt are referencing other requirements.txt, the dependency resolution gets messy.
6. ReadTheDocs has simple, elegant support for specifically including select extras.
7. As mentioned above, it's nice to be able to declare most or all of a packages dependencies in one place. It makes it easy to see the broader scope of dependencies a package uses.

Although the documentation doesn't explicitly mention these use cases, I think that's because extras were designed to be very general-purpose - to allow a package to declare sets of optional dependencies by name. It's their generality and simple elegance that's enabled them to be readily used by tools like tox and rtd. But based on the experiences above, both before and after, I'm very much pleased with the use of extras for declaring tests and docs dependencies.

So yes, I do endorse the idea.

But addressing more specifically the idea that pip should have support for --deps-only, this is a feature I've been thinking about ticketing myself. And apparently it's been ticketed before in #2171, which may be why I haven't ticketed it.

At the moment, I can't recall the use-cases that led me to want to install the deps only, but I can think of one right now: You want to get a shell for a package you have checked out, but you don't want to create a virtualenv nor install the dependencies into your current env, you could use rwt, except rwt uses pip install -t which doesn't work with -e. Which for many packages wouldn't be an issue because the source for this current package may be simply importable as-is (without any installation assistance).

Such a feature may also be helpful in troubleshooting--in being able to bisect an issue, first ensuring that dependencies install correctly, then separately testing that the package itself installs. I'm sure that would have been useful when troubleshooting race conditions.

Unless there's something about this feature that would unduly complicate pip, I'm enthusastically +1.

[gaborbernat]

@pfmoore I'm still excited about this happening, what's your view on this nowadays?

[pfmoore]

I remain unclear on the value of this (disclaimer: I basically never use extras myself) but that's all. Nothing's going to happen without a PR, and if one is submitted, I'll probably just leave it to the other pip developers (who may well have stronger opinions than I do) to review/debate it. Nothing's blocked on me, that I know of.

[gaborbernat]

@pfmoore how do you keep separate requirements for various tasks - e.g. tests, documentation, pylint, mypy, etc? or do you just put all in a global requirements.txt and hope there's no conflict in between them?

[pfmoore]

Multiple requirements files (see pip itself) or simply have projects small enough that I can handle it all manually :-) (My day to day use of Python is adhoc data analysis, automation scripts and personal tools, not so much large project development - I'm likely not a good example of someone who'd find this feature useful, is all).

[gaborbernat]

In that case, all I can say is that the value added with this would be having one file (the setup.py which already requires to define install dependencies) to list all dependencies (either install, testing, documentation generation, etc). Shipping this one file via sdist is also solved automatically (as extras will be part of the package metadata).

[pradyunsg]

@lkollar is working on a POC PR for this.

[lkollar]

I think a very important use case hasn't been mentioned on this issue yet (and the main reason why I implemented it in #5950): it makes it possible to install the build and test dependencies and then independently install and build the sdist, and run the tests against it. An example where this is needed is a CI pipeline where --deps-only with extras_require makes it easy to validate the sdist.

I think in order to fully support the extras_require based workflow this flag is necessary. I'd like to fully implement @brettcannon's recommendation for dependency management for libraries, but without --deps-only this is not feasible.

Any feedback on the PR would be appreciated.

[gaborbernat]

I no longer think this would be a good Idea so closing this for now.

[akaihola]

Another use case would have been writing a Dockerfile with dependencies and the application/library itself in separate layers. This would allow for iterating quicker when developing an app/library which has

• dependencies that take a long time to install (lots of them, large ones, ones with C/Cython extensions)
• require local testing in a container during development

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.