Roadmap, short- to medium-term

[jherland]

A high-level collection of tasks/issues that we should be able to tackle fairly soon, and that would significantly improve the user experience of FawltyDeps:

Improving our matching of dependencies vs imports

• Finding and fixing various corner cases where our mapping from dependency names to import names using the local environment is incorrect.
  • Related issues: Dependency to import name(s) mapping from local environment: a roadmap #142, typing-extensions vs. typing_extensions #173, Bug: Dependency to Import mapping on the qiskit package does not include the qiskit import name #176, Packages that are installed from local (editable?) folders are not reported by packages_distributions()? #187
• Teach FawltyDeps to look at a different environment than what it itself is running in. This would allow you to e.g. pipx install fawltydeps to have it available across all your projects instead of having to install it in every project's own virtualenv.
  • Related issues: Add the possibility to point towards the path of the environment to use for inferring package distribution information. #161
• Look at fully decoupling ourselves from the local environment: pull information about packages and their import names directly from PyPI? Maybe include a hardcoded registry of common cases inside FawltyDeps itself?
  • Related issues: Investigate possibility of including modules exposed by packages in PyPI #32
  • See also the design document

Parsing imports and dependencies

• A generic and easy-to-use directive for ignoring files.
  • Along the same lines as when we added the basepath positional argument (which is a shortcut for passing --code and --deps) we should have a --ignore(?) option for easily ignoring parts of the directory tree, both with respect to parsing code and deps.
  • Do we want to ignore some things by default? build/? dist/? virtualenvs? __pycache__? Maybe safer to reuse stuff from .gitignore?
  • Related issue: Exclude directories from analysis #226
• Improve our imports parsing:
  • Try to recognize and parse some common uses of importlib for importing modules.
    • Related issues: Extract imports from function calls #46
  • At least warn the user when we see that a dynamic import is likely happening, but we're unable to fully parse it.
  • Improve our detection of sys.path and similar environment manipulation, and warn the user of how this likely affects our classification of 1st- vs 3rd-party packages.
  • Add knobs to allow the user to tell us where more 1st-party packages can be found.
  • Recognize the context in which something is imported, and how that should affect our reporting (see issue Learn from Pants #171 for inspiration). Recognize e.g. the following patterns:
    • try: import ... except ImportError: ...
    • if sys.version_info >= ...: import ... else: import ...
  • Deal with various corner cases of parsing notebooks, e.g. Magic commands (episode 138): in the middle of a line #132
• Improve our dependency parsing:
  • Are there more dependency formats that we need to support?
    • What about Conda?
    • Related issues: Support parsing dependencies from Flit pyproject.toml dependency fields #49, Support parsing dependencies from setuptools pyproject.toml dependency fields #50, Build back-ends with possibly specific pyproject.toml dependency metadata #51, Support even more file types for declaring dependencies? #66
  • Deal with various corner cases, e.g. Requirements files may refer to other requirements files #113, Line continuations aren't fully supported for per-requirement options in requirements.txt #225
  • Parse dependencies that can be considered "declared" via magic commands in notebooks, e.g. Parsing dependencies declared in magic commands in Jupyter notebooks #92
• How to deal with namespace packages? Support namespace packages  #45

Presenting/filtering results

• Differentiate between dependency groups (specifically main/mandatory deps vs. extras/dev/test/optional deps), and allow the user to specify which deps FD should check.
  • Related issues: Different reporting behavior for "core" and "extra" dependencies #44
• Better handling of tools and similar supplemental packages, that is, dependencies that are declared, but never meant to be imported directly.
  • Examples:
    • CLI tools never meant to be imported: How to handle tools (not imported) that end up being reported as "unused" dependencies #117
    • Supplemental packages that provide type annotations for better type checking: Detect dependencies used for types #221
    • Supplemental packages that provide functionality for a direct dependency: Unable to detect BeautifulSoup html5lib parser #246
  • Do we carry a hardcoded list of common tools in the Python ecosystem?
  • Do we e.g. ignore all types-* packages by default?
  • Do we check the project configuration for evidence of a tool being used before auto-ignoring it?
• Handle conditional imports better:
  • Some conditional imports are clear-cut and probably don't require optionality: Imports protected by TYPE_CHECKING are treated as mandatory #390
  • Other conditional imports need more analysis, and probably an option to include/exclude from analysis?
  • Conditional imports vs. Alternative imports?