Environment dependency locking

[ofek]

Planning for when virtualenv upgrades pip to a version that includes pypa/pip#10771

I'm just documenting my thoughts at this point 🙂

Environments

Environment plugins will gain 2 new methods:

def read_file(self, relative_path: str) -> str:
    ...

def write_file(self, relative_path: str, contents: str):
    ...

Dependency lockers

Dependency lockers will become a new type of plugin:

def generate(self, dependencies: list[str]):
    """
    The input is a list of user-defined PEP 508 strings.
    """
    lock = self.do(dependencies)
    self.environment.write_file('...', lock)

def in_sync(self, dependencies: list[str]) -> bool:
    """
    The input is a list of user-defined PEP 508 strings.
    """
    lock = self.do(dependencies)
    return self.environment.read_file('...') == lock

The environment attribute will be a minimal wrapper over an instance of EnvironmentInterface:

def read_file(self, relative_path: str) -> str:
    return self.__env.read_file(relative_path)

def write_file(self, relative_path: str, contents: str):
    self.__env.write_file(relative_path, contents)

def run_shell_command(self, command: str, **kwargs) -> subprocess.CompletedProcess:
    with self.__env.command_context():
        return self.__env.run_shell_command(command, **kwargs)

Commands

• hatch dep ...
  • lock - ...

Usage

...

[ashemedai]

@ofek Looks to be virtualenv 20.16.0 since 26th of July.

[bodograumann]

I would love to switch to hatch, given all the pain poetry has caused me lately.
Having the feature to lock down dependencies would be a prerequisite though.

[maresb]

@bodograumann, would #522 work for you?

[bodograumann]

Not exactly.
Dependency locking is not only about fixing a dependency version out of the given range.
More importantly locking should store and later verify the content hashes of the packages.

What I was hoping for was something along the lines of what deno does:

• Run a lock command to generate the lock file to commit. This should be possible with pip install --dry-run --report, from what I understood.
• Check the installed dependencies against the lock file. It might be possible to have a script that again generates the pip dependency report and diffs that against the committed one, but draging that script around everywhere seems inconvenient.

Having a way to install the exact dependency versions from the lock file might be nice, but we could simply use exact versions in the pyproject.toml instead. This is the common practice in deno I think.

[maresb]

More importantly locking should store and later verify the content hashes of the packages.

Have you seen the --generate-hashes flag to pip-compile? It may be able to do what you want. For example,

pip-compile --generate-hashes --output-file=requirements.out pyproject.toml

Check the installed dependencies against the lock file. It might be possible to have a script that again generates the pip dependency report and diffs that against the committed one, but draging that script around everywhere seems inconvenient.

Does pip-sync do what you need?

Having a way to install the exact dependency versions from the lock file might be nice, but we could simply use exact versions in the pyproject.toml instead. This is the common practice in deno I think.

The general recommendation seems to be putting flexible versions in pyproject.toml and lock that to exact versions in a requirements.txt.

[bodograumann]

pip-compile --generate-hashes and pip-sync seem to do what we need, but how could that work when hatch already manages the dependencies?
Assuming I generate a requirements.txt with pip-compile, open a virtual environment with hatch and install the correct locked dependencies. What happens when I now do hatch run …. Will it keep the installed dependencies in the virtual environment as they are even when newer versions would be available to install?

I tried executing this in an example project, but pip-sync uninstalls system and user packages (and fails because it is not root).
How can I tell it to only consider the current virtual environment?

[maresb]

Thanks for clarifying. This is a very interesting question. I wonder if it would be possible to make an environment plugin so that hatch and pip-tools cooperate.