-
Notifications
You must be signed in to change notification settings - Fork 626
Python
Python 3.3 and later provide built-in support for virtual environments via the venv module in the standard library. This means the virtualenv
tool is no longer needed.
For direnv v2.21.0 or later, The default python layout uses venv to sandbox a project's dependencies.
Add this to the .envrc
:
layout python
The first time the .envrc
is loaded it will automatically create the virtualenv under .direnv/python-$python_version
. The sandbox is also automatically activated whenever direnv loads the .envrc
file (although the prompt won't change by default, however see here or here).
On direnv v2.32.1
, you can specify a different path for virtualenv:
export VIRTUAL_ENV=venv
layout python
For versions earlier than v2.21.0, the default python layout uses virtualenv to sandbox a project's dependencies.
If you want to use venv
alternative of virtualenv
, add this snippet to your ~/.config/direnv/direnvrc
:
realpath() {
[[ $1 = /* ]] && echo "$1" || echo "$PWD/${1#./}"
}
layout_python-venv() {
local python=${1:-python3}
[[ $# -gt 0 ]] && shift
unset PYTHONHOME
if [[ -n $VIRTUAL_ENV ]]; then
VIRTUAL_ENV=$(realpath "${VIRTUAL_ENV}")
else
local python_version
python_version=$("$python" -c "import platform; print(platform.python_version())")
if [[ -z $python_version ]]; then
log_error "Could not detect Python version"
return 1
fi
VIRTUAL_ENV=$PWD/.direnv/python-venv-$python_version
fi
export VIRTUAL_ENV
if [[ ! -d $VIRTUAL_ENV ]]; then
log_status "no venv found; creating $VIRTUAL_ENV"
"$python" -m venv "$VIRTUAL_ENV"
fi
PATH="${VIRTUAL_ENV}/bin:${PATH}"
export PATH
}
Now you can use it like this in your .envrc
:
layout python-venv
To specify the Python executable to use, use:
layout python-venv python3.6
(This works similar to the layout python ...
that ships with direnv
itself.)
To use a different directory for the virtualenv, set the $VIRTUAL_ENV
variable to the desired path, which may be relative:
export VIRTUAL_ENV=.venv
layout python-venv python3.6
The $PS1
is not modified during activation as it does when doing it by hand. To restore that functionality it's possible to add something like this in your .zshrc/.bashrc
. See also this page for an alternative solution.
.bashrc
show_virtual_env() {
if [[ -n "$VIRTUAL_ENV" && -n "$DIRENV_DIR" ]]; then
echo "($(basename $VIRTUAL_ENV))"
fi
}
export -f show_virtual_env
PS1='$(show_virtual_env)'$PS1
If you do not add this export show_virtual_env
line, when you entering a sub-bash process, the function will not be seen and some error will occur.
.zshrc
setopt PROMPT_SUBST
show_virtual_env() {
if [[ -n "$VIRTUAL_ENV" && -n "$DIRENV_DIR" ]]; then
echo "($(basename $VIRTUAL_ENV))"
fi
}
PS1='$(show_virtual_env)'$PS1
the setopt PROMPT_SUBST
turns on substitution of the prompt string. See man zshell
.
If using Conda instead of virtualenv, replace $VIRTUAL_ENV
with $CONDA_DEFAULT_ENV
.
If using pyenv to switch between python/conda you can rewrite show_virtual_env()
to
check if pyenv local is a *conda*
-version.
show_virtual_env() {
if [[ $(pyenv local 2>/dev/null) == *"conda"* ]]; then
VENV=$CONDA_DEFAULT_ENV
else
VENV=$VIRTUAL_ENV
fi
if [[ -n "$VENV" && -n "$DIRENV_DIR" ]]; then
echo "($(basename $VENV))"
fi
}
The layout python
directive will look for a python
executable on the path. It's possible to specify another executable if the virtualenv needs to be created with a different one. For example: layout python ~/.pyenv/versions/2.7.6/bin/python
For python3 there is a shortcut layout python3
To use pyenv and venv (falling back to virtualenv for Python < 3.3) to create and load a virtual environment under $PWD/.direnv/python-$python_version
, add the following to .envrc
:
layout pyenv 3.6.7
layout_pyenv
was first added to direnv v2.21.0. If you are using an older version of direnv, you'll need to add the following to ~/.config/direnv/direnvrc
or ~/.direnvrc
:
layout_pyenv() {
unset PYENV_VERSION
# Because each python version is prepended to the PATH, add them in reverse order
for ((j = $#; j >= 1; j--)); do
local python_version=${!j}
local pyenv_python=$(pyenv root)/versions/${python_version}/bin/python
if [[ ! -x "$pyenv_python" ]]; then
log_error "Error: $pyenv_python can't be executed."
return 1
fi
unset PYTHONHOME
local ve=$($pyenv_python -c "import pkgutil; print('venv' if pkgutil.find_loader('venv') else ('virtualenv' if pkgutil.find_loader('virtualenv') else ''))")
case $ve in
"venv")
VIRTUAL_ENV=$(direnv_layout_dir)/python-$python_version
export VIRTUAL_ENV
if [[ ! -d $VIRTUAL_ENV ]]; then
$pyenv_python -m venv "$VIRTUAL_ENV"
fi
PATH_add "$VIRTUAL_ENV"/bin
;;
"virtualenv")
layout_python "$pyenv_python"
;;
*)
log_error "Error: neither venv nor virtualenv are available to ${pyenv_python}."
return 1
;;
esac
# e.g. Given "use pyenv 3.6.9 2.7.16", PYENV_VERSION becomes "3.6.9:2.7.16"
[[ -z "${PYENV_VERSION-}" ]] && PYENV_VERSION=$python_version || PYENV_VERSION="${python_version}:$PYENV_VERSION"
done
export PYENV_VERSION
}
Notably, this method does not depend on pyenv-virtualenv and therefore follows layout_python
's convention of creating the virtual environment under $PWD/.direnv/python-$python_version
rather than under $(pyenv root)/versions/$python_version/envs/
.
It's possible to use pyenv-virtualenv to manage python versions and virtualenvs, and rely on direnv to load them.
For this add the following in the ~/.direnvrc
(or ~/.config/direnv/direnvrc
) file:
### use a certain pyenv version
use_python() {
if [ -n "$(which pyenv)" ]; then
local pyversion=$1
pyenv local ${pyversion}
fi
}
layout_virtualenv() {
local pyversion=$1
local pvenv=$2
if [ -n "$(which pyenv virtualenv)" ]; then
pyenv virtualenv --force --quiet ${pyversion} ${pvenv}-${pyversion}
fi
pyenv local --unset
}
layout_activate() {
if [ -n "$(which pyenv)" ]; then
source $(pyenv root)/versions/$1/bin/activate
fi
}
Using pyenv, install a couple of versions.
Then in any project's .envrc
:
# -*- mode: sh; -*-
# (rootdir)/.envrc : direnv configuration file
# see https://direnv.net/
# pyversion=$(head .python-version)
# pvenv=$(head .python-virtualenv)
pyversion=2.7.14
pvenv=myproject
use python ${pyversion}
# Create the virtualenv if not yet done
layout virtualenv ${pyversion} ${pvenv}
# activate it
layout activate ${pvenv}-${pyversion}
You can replace your myproject above with: pvenv=$(basename $PWD)
to default to the base name of the current path. I use a BASH function to drop the contents of this into the .envrc in $CWD so I have less setup/editing files.
Instead of running layout activate ${pvenv}-${pyversion}
as shown above, you can run export PYENV_VERSION=${pvenv}-${pyversion}
, which tells pyenv to use the virtualenv via shims. Useful if you want pyenv
to be aware of the active virtualenv when you run commands such as pyenv versions
.
You can define a new layouts in your ~/.direnvrc
file to activate a virtual environment automatically.
Add this to your ~/.direnvrc
file:
layout_virtualenv() {
local venv_path="$1"
source ${venv_path}/bin/activate
# https://github.com/direnv/direnv/wiki/PS1
unset PS1
}
layout_virtualenvwrapper() {
local venv_path="${WORKON_HOME}/$1"
layout_virtualenv $venv_path
}
and use it like this in .envrc
of project folder:
layout virtualenv /path/to/my-awesome-project
or
layout virtualenvwrapper my-awesome-project
$ echo layout pipenv >> .envrc
Or more generally, in the .envrc
:
layout pipenv
It's possible to use anaconda for virtual environments.
For this add the following in the ~/.direnvrc
(or ~/.config/direnv/direnvrc
) file:
layout_anaconda() {
local ACTIVATE="${HOME}/miniconda3/bin/activate"
if [ -n "$1" ]; then
# Explicit environment name from layout command.
local env_name="$1"
source $ACTIVATE ${env_name}
elif (grep -q name: environment.yml); then
# Detect environment name from `environment.yml` file in `.envrc` directory
source $ACTIVATE `grep name: environment.yml | sed -e 's/name: //' | cut -d "'" -f 2 | cut -d '"' -f 2`
else
(>&2 echo No environment specified);
exit 1;
fi;
}
Then specify anaconda in your .envrc
with:
layout anaconda root
or
layout anaconda
to activate an environment specified in environment.yml
.
Note: if you have installed Anaconda via brew
, you might need to amend the local ANACONDA_HOME
definition to this:
local ANACONDA_HOME=/usr/local/miniconda3
Using direnv with Pycharm explained...
You can use direnv to create your local virtualenv following normal means:
echo "layout python3" > .envrc
But certain editors that look for local .env
, env
or venv
will not find .direnv/python-x.x.x/. You can however create a soft link. Pycharm will automatically find and use the local environment.
echo "ln -s .direnv/\$(basename \$VIRTUAL_ENV)/ .env" >> .envrc
Shared my entire setup for this in a gist.
Similar to layout_python
, but uses poetry
to build a virtualenv from the pyproject.toml
located in the same directory.
Add the following to ${XDG_CONFIG_HOME:-${HOME}/.config}/direnv/direnvrc
. This has been submitted as PR#995 for consideration to have poetry support built into the direnv
stdlib.
layout_poetry() {
PYPROJECT_TOML="${PYPROJECT_TOML:-pyproject.toml}"
if [[ ! -f "$PYPROJECT_TOML" ]]; then
log_status "No pyproject.toml found. Executing \`poetry init\` to create a \`$PYPROJECT_TOML\` first."
poetry init
fi
if [[ -d ".venv" ]]; then
VIRTUAL_ENV="$(pwd)/.venv"
else
VIRTUAL_ENV=$(poetry env info --path 2>/dev/null ; true)
fi
if [[ -z $VIRTUAL_ENV || ! -d $VIRTUAL_ENV ]]; then
log_status "No virtual environment exists. Executing \`poetry install\` to create one."
poetry install
VIRTUAL_ENV=$(poetry env info --path)
fi
PATH_add "$VIRTUAL_ENV/bin"
export POETRY_ACTIVE=1 # or VENV_ACTIVE=1
export VIRTUAL_ENV
}
For the poetry init
line it could be improved further with something like the following:
# Specify the active python (major.minor version) rather than the one used to install poetry in case they are not the same.
poetry init --python ^$(python3 --version 2>/dev/null | cut -d' ' -f2 | cut -d. -f1-2)
We could also include the --no-interaction
option to simply make the pyproject.toml
with the default suggestions and avoid direnv
reporting the process is taking too long.
When entering a directory where you have layout_poetry
set, poetry env info
can be slow to invoke. Poetry has a setting called virtualenvs.in-project, which creates the .venv
directory within your project. So we check if a .venv
directory exists already, before invoking poetry env info
.
Workflow for a new project, new-project
say:
-
poetry new --src new-project
where the--src
flag is optional. This creates thepyproject.toml
file. cd new-project
- Create an
.envrc
file withecho 'layout poetry' > .envrc
-
direnv allow
...poetry install
executes creating the virtual environment, adding the virtual environment python version to your$PATH
and finally activating the virtual environment setting the ENV variables$VIRTUAL_ENV
and$POETRY_ACTIVE
as we expect. - Start to work
Workflow for an existing project, existing-project
say:
cd existing-project
- Create an
.envrc
file withecho 'layout poetry' > .envrc
-
direnv allow
...poetry init
executes interactively creating thepyproject.toml
...poetry install
executes creating the virtual environment, adding the virtual environment python version to your$PATH
and finally activating the virtual environment setting the ENV variables$VIRTUAL_ENV
and$POETRY_ACTIVE
as we expect. - Start to work
Similar to layout_python
, but uses PDM
to build a virtualenv from the pyproject.toml
located in the same directory.
layout_pdm() {
PYPROJECT_TOML="${PYPROJECT_TOML:-pyproject.toml}"
if [ ! -f "$PYPROJECT_TOML" ]; then
log_status "No pyproject.toml found. Executing \`pmd init\` to create a \`$PYPROJECT_TOML\` first."
pdm init --non-interactive --python "$(python3 --version 2>/dev/null | cut -d' ' -f2 | cut -d. -f1-2)"
fi
VIRTUAL_ENV=$(pdm venv list | grep "^\*" | awk -F" " '{print $3}')
if [ -z "$VIRTUAL_ENV" ] || [ ! -d "$VIRTUAL_ENV" ]; then
log_status "No virtual environment exists. Executing \`pdm info\` to create one."
pdm info
VIRTUAL_ENV=$(pdm venv list | grep "^\*" | awk -F" " '{print $3}')
fi
PATH_add "$VIRTUAL_ENV/bin"
export PDM_ACTIVE=1 # or VENV_ACTIVE=1
export VIRTUAL_ENV
}
Workflow for a new project, new-project
say:
mkdir new-project
cd new-project
- Create an
.envrc
file withecho 'layout pdm' > .envrc
-
direnv allow
...pdm init
executes creating thepyproject.toml
...pdm info
executes adding the virtual environment python version to your$PATH
and finally activating the virtual environment setting the ENV variables$VIRTUAL_ENV
and$PDM_ACTIVE
as we expect. - Start to work
Workflow for an existing project, existing-project
say:
cd existing-project
- Create an
.envrc
file withecho 'layout pdm' > .envrc
-
direnv allow
...pdm init
executes creating thepyproject.toml
...pdm info
executes creating the virtual environment, adding the virtual environment python version to your$PATH
and finally activating the virtual environment setting the ENV variables$VIRTUAL_ENV
and$PDM_ACTIVE
as we expect. - Start to work
Similar to layout_python
, but uses hatch
to build a virtualenv from the pyproject.toml
located in the same directory.
Add the following to ${XDG_CONFIG_HOME:-${HOME}/.config}/direnv/direnvrc
.
layout_hatch() {
if [[ ! -f "pyproject.toml" ]]; then
if [[ ! -f "setup.py" ]]; then
local tmpdir
log_status "No pyproject.toml or setup.py found. Executing \`hatch new\` to create a new project."
PROJECT_NAME=$(basename $PWD)
tmpdir="$(mktemp -d)"
hatch new $PROJECT_NAME $tmpdir > /dev/null
cp -a --no-clobber $tmpdir/* . && rm -rf $tmpdir
else
# I haven't yet seen a case where migrating from an existing `setup.py` works, but I'm sure there are some.
log_status "No pyproject.toml found. Executing \`hatch new --init\` to migrate from setuptools."
hatch new --init || log_error "Failed to migrate from setuptools. Please fix and run \`hatch new --init\` manually." && return 1
fi
fi
HATCH_ENV=${HATCH_ENV_ACTIVE:-default}
# We need this to error out if the env doesn't exist in the pyproject.toml file.
VIRTUAL_ENV=$(hatch env find $HATCH_ENV)
if [[ ! -d $VIRTUAL_ENV ]]; then
log_status "No virtual environment exists. Executing \`hatch env create\` to create one."
hatch env create $HATCH_ENV
fi
PATH_add "$VIRTUAL_ENV/bin"
export HATCH_ENV_ACTIVE=$HATCH_ENV # or VENV_ACTIVE=1
export VIRTUAL_ENV
}
-
hatch
doesn't seem to use$PYPROJECT_TOML
, so it's hard-coded. - Initializing from an existing
setup.py
is supported, but doesn't always work.
The workflow for a new project, say, new-project
would be:
-
hatch new [--cli] new-project
This creates a project layout (including apyproject.toml
file) as configured in your${XDG_CONFIG_HOME:-${HOME}/.config}/hatch/config.toml
file.
-
--cli
adds an optional dependency onclick
.
cd new-project
- Create an
.envrc
file withecho 'layout hatch' > .envrc
-
direnv allow
…hatch env create
executes creating the virtual environment, adding the virtual environment python version to your$PATH
, and finally activating the virtual environment, setting the ENV variables$VIRTUAL_ENV
andHATCH_ENV_ACTIVE
as we expect. - Start to work
For an existing project, say, existing-project
the workflow would be:
cd existing-project
- Create an
.envrc
file withecho 'layout hatch' > .envrc
-
direnv allow
…
- If there is a
setup.py
file,hatch new --init
executes, creating thepyproject.toml
-
hatch env create
executes, creating the virtual environment, adding the virtual environment python version to your$PATH
, and finally activating the virtual environment setting the ENV variables$VIRTUAL_ENV
and$HATCH_ENV_ACTIVE
as we expect.
- Start to work
Similar to layout_python
, but uses rye
to build a virtualenv from the pyproject.toml
located in the same directory.
Add the following to ${XDG_CONFIG_HOME:-${HOME}/.config}/direnv/direnvrc
.
layout_rye() {
PYPROJECT_TOML="${PYPROJECT_TOML:-pyproject.toml}"
if [[ ! -f "$PYPROJECT_TOML" ]]; then
log_status "No pyproject.toml found. Executing \`rye init\` to create a \`$PYPROJECT_TOML\` first."
rye init
fi
if [[ -d ".venv" ]]; then
VIRTUAL_ENV="$(pwd)/.venv"
fi
if [[ -z $VIRTUAL_ENV || ! -d $VIRTUAL_ENV ]]; then
log_status "No virtual environment exists. Executing \`rye sync\` to create one."
rye sync
VIRTUAL_ENV="$(pwd)/.venv"
fi
PATH_add "$VIRTUAL_ENV/bin"
export RYE_ACTIVE=1 # or VENV_ACTIVE=1
export VIRTUAL_ENV
}
Workflow for a new project, new-project
say:
-
rye init new-project
. This creates thepyproject.toml
file. cd new-project
- Create an
.envrc
file withecho 'layout rye' > .envrc
-
direnv allow
...rye sync
executes creating the virtual environment, adding the virtual environment python version to your$PATH
and finally activating the virtual environment setting the ENV variables$VIRTUAL_ENV
and$RYE_ACTIVE
as we expect. - Start to work
Workflow for an existing project, existing-project
say:
cd existing-project
- Create an
.envrc
file withecho 'layout rye' > .envrc
-
direnv allow
...rye init
executes creating thepyproject.toml
...rye sync
executes creating the virtual environment, adding the virtual environment python version to your$PATH
and finally activating the virtual environment setting the ENV variables$VIRTUAL_ENV
and$RYE_ACTIVE
as we expect. - Start to work
Similar to layout_python
, but uses uv
to build a virtualenv.
Add the following to ${XDG_CONFIG_HOME:-${HOME}/.config}/direnv/direnvrc
.
layout_uv() {
if [[ -d ".venv" ]]; then
VIRTUAL_ENV="$(pwd)/.venv"
fi
if [[ -z $VIRTUAL_ENV || ! -d $VIRTUAL_ENV ]]; then
log_status "No virtual environment exists. Executing \`uv venv\` to create one."
uv venv
VIRTUAL_ENV="$(pwd)/.venv"
fi
PATH_add "$VIRTUAL_ENV/bin"
export UV_ACTIVE=1 # or VENV_ACTIVE=1
export VIRTUAL_ENV
}
Workflow for a new project, new-project
say:
mkdir new-project
cd new-project
- Create an
.envrc
file withecho 'layout uv' > .envrc
-
direnv allow
...uv venv
executes creating the virtual environment, adding the virtual environment python version to your$PATH
and finally activating the virtual environment setting the ENV variables$VIRTUAL_ENV
and$UV_ACTIVE
as we expect. - Start to work
Workflow for an existing project, existing-project
say:
cd existing-project
- Create an
.envrc
file withecho 'layout uv' > .envrc
-
direnv allow
... if .venv does not already existuv venv
executes creating the virtual environment, adding the virtual environment python version to your$PATH
and finally activating the virtual environment setting the ENV variables$VIRTUAL_ENV
and$UV_ACTIVE
as we expect. - Start to work