Ignore literal duplicates (includes and/or individual requirements)

[rfleschenberg]

pip's handling of requirements files does not support a scenario like this one:

# common.txt
somepackage==1.0

# testing.txt
- r common.txt
test-tool==1.0

# development.txt
-r common.txt
-r testing.txt
developer-tool==1.0

Here, pip install -r development.txt will complain about a double requirement somepackage==1.0 and quit.

Because of that, I have to duplicate my testing requirements in my development.txt. Not a big deal, but it would be nice if pip could handle this. For this scenario, it seems that it would be sufficient to just ignore the duplicate -r line, so this would probably be easier to implement than #56.

[robertfw]

Just ran into the exact same issue, same use case.

[carlosviol]

+1

[qwcode]

closing as dupe of #56

[qwcode]

reopening, this is not a duplicate of #56. this issue is about literal duplicate includes, whereas #56 is about resolving non-conflicting (but not necessarily duplicate) top-level requirements.

[jgumbley]

+1

[qwcode]

I looked at this from the angle of attempting to allow/ignore literal duplicate requirements.

One snag was with non-editable url requirements, e.g. pkg urls or vcs urls without -e.
currently, that code path does't store the full url in the url property of InstallRequirement, but rather just the url without the fragment.

e.g. so there could be 2 lines like this

git+https://project.git@branch#egg=project'
git+https://project.git@branch#egg=anotherproject'

I don't think these 2 should pass thru as literal duplicates, but we don't have enough information stored to distinguish them. we should look into storing the full url I think, and understand why we're not. I'm not sure atm

related issue: #1658

[dtheodor]

👍

Pip is more disappointing the more you try to do anything outside of tutorial material.

[kaustavha]

+1 same error similiar usecase.
Related issue: rackerlabs/mimic#502