hatch test fails for packages that install executables in the virtualenv's bin

[lanzz]

I'm developing an application that has bottle as an indirect dependency (via pywebview), and bottle installs a bottle.py script in the virtual environment's bin directory. When I run hatch test, it apparently executes with the virtual environment's bin as current directory, so import bottle in the app mistakenly gets the bin/bottle.py script instead of the bottle module installed in the library directory. For some reason that also raises an exception about bottle.py containing an from __future__ import ... statement not at the top of the file (this seems to be caused by some sort of a shim at the top of the file that runs it indirectly through /bin/sh, which seems to be added by hatch — it is only present in the hatch-test.py3.12 environment but not in the default environment), but I believe the main problem is importing the wrong bottle.py in the first place. I think it's a fairly typical pattern for packages to install a script in the bin directory that matches the name of the package, so this behaviour is probably not limited to bottle.

$ hatch test
================================================================================================================ test session starts ================================================================================================================
platform darwin -- Python 3.12.2, pytest-8.2.2, pluggy-1.5.0
rootdir: /Users/lanzz/Projects/edva
configfile: pyproject.toml
plugins: rerunfailures-14.0, anyio-4.4.0, mock-3.14.0, xdist-3.6.1
collected 0 items / 1 error                                                                                                                                                                                                                         

====================================================================================================================== ERRORS =======================================================================================================================
_________________________________________________________________________________________________________ ERROR collecting tests/test_ui.py _________________________________________________________________________________________________________
../../Library/Application Support/hatch/env/virtual/edva/WQlRx7L0/hatch-test.py3.12/lib/python3.12/site-packages/_pytest/python.py:492: in importtestmodule
    mod = import_path(
../../Library/Application Support/hatch/env/virtual/edva/WQlRx7L0/hatch-test.py3.12/lib/python3.12/site-packages/_pytest/pathlib.py:591: in import_path
    importlib.import_module(module_name)
/opt/homebrew/Cellar/[email protected]/3.12.2_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/importlib/__init__.py:90: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1387: in _gcd_import
    ???
<frozen importlib._bootstrap>:1360: in _find_and_load
    ???
<frozen importlib._bootstrap>:1331: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:935: in _load_unlocked
    ???
../../Library/Application Support/hatch/env/virtual/edva/WQlRx7L0/hatch-test.py3.12/lib/python3.12/site-packages/_pytest/assertion/rewrite.py:178: in exec_module
    exec(co, module.__dict__)
tests/test_ui.py:5: in <module>
    from edva.ui import UI, main
src/edva/ui.py:15: in <module>
    import webview  # type: ignore[import-untyped]  # nothing we can until it's fixed upstream
../../Library/Application Support/hatch/env/virtual/edva/WQlRx7L0/hatch-test.py3.12/lib/python3.12/site-packages/webview/__init__.py:25: in <module>
    import webview.http as http
../../Library/Application Support/hatch/env/virtual/edva/WQlRx7L0/hatch-test.py3.12/lib/python3.12/site-packages/webview/http.py:31: in <module>
    import bottle
E     File "/Users/lanzz/Library/Application Support/hatch/env/virtual/edva/WQlRx7L0/hatch-test.py3.12/bin/bottle.py", line 18
E       from __future__ import with_statement
E       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E   SyntaxError: from __future__ imports must occur at the beginning of the file
============================================================================================================== short test summary info ==============================================================================================================
ERROR tests/test_ui.py
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
================================================================================================================= 1 error in 0.10s ==================================================================================================================

As you can see, the exception is raised in /Users/lanzz/Library/Application Support/hatch/env/virtual/edva/WQlRx7L0/hatch-test.py3.12/bin/bottle.py, which is the script installed in the virtualenv bin, but the bottle module itself is installed as /Users/lanzz/Library/Application Support/hatch/env/virtual/edva/WQlRx7L0/hatch-test.py3.12/lib/python3.12/site-packages/bottle.py (which doesn't contain the shim that breaks the future import).

The shim at the start of bin/bottle.py:

#!/bin/sh
'''exec' '/Users/lanzz/Library/Application Support/hatch/env/virtual/edva/WQlRx7L0/hatch-test.py3.12/bin/python' "$0" "$@"
' '''
# -*- coding: utf-8 -*-
(the normal content of bottle.py follows)

I have no idea why that shim is there in the first place. The lib module starts with a more typical #!/usr/bin/env python.