An extremely simplified interface is provided to accomplish all of the following with no dependencies outside dbt-core:
-
Parse dbt project on disk loading dbt core classes into memory from a single class/interface
-
Automatic management of the adapter and thread-safe efficient connection pool reuse
-
Run SQL and get results in python fully independent of the dbt adapter which automatically enables support for many databases
-
Run SQL with dbt SQL from a single method call
-
Load macros at runtime enabling custom functionality in third party extensions without requiring the dbt packaging system to be managed in userland
-
Compile dbt jinja extremely fast and efficiently, thread-safe and stress tested at load via a Bottle server which live compiles SQL
-
Manage multiple dbt projects in a single process using the DbtProjectContainer class
dbt-core-interface
is a wrapper that allows developers to rapidly develop features and integrations for dbt. This project aims to serve as a place for the community to aggregate the best ways to interface with dbt. It is afforded a much faster iteration cycle and much more freedom due to it's independence from the dbt codebase. It is intended to act as an common library to dbt's existing APIs for developers. Implementations can land here and prove themselves out before landing in the dbt-core codebase and benefit all developers involved. Sqlfluff dbt templater, dbt-osmosis, dbt-fastapi which I am ripping out of dbt-osmosis, an impending metadata manager, a testing framework will all leverage this library. As dbt core evolves and stabilizes its python API, this project will evolve with it. This may manifest in simplification of certain methods but our goal is to maintain the API and focus on driving efficient, innovative/creative, and agile community driven integration patterns.
- The only requirement is dbt-core, tested with versions
1.0.*
,1.1.*
,1.2.*
,1.3.*
You can install Dbt Core Interface via pip from PyPI:
$ pip install dbt-core-interface
Please see the Api Reference for details.
To launch the Bottle server for live compiling dbt jinja:
python -m dbt_core_interface.project
This will launch the server on port 8581. You can then make requests to the server, e.g.:
curl -X POST -H "Content-Type: application/json" -H "X-dbt-Project: dbt_project" -d '{"project_dir":"/app/tests/sqlfluff_templater/fixtures/dbt/dbt_project/","profiles_dir":"/app/tests/sqlfluff_templater/fixtures/dbt/profiles_yml/","target":"dev"}' https://localhost:8581/register
You can change the server hostname and port using the --host
and --port
arguments.
Contributions are very welcome. To learn more, see the Contributor Guide.
Distributed under the terms of the MIT license, Dbt Core Interface is free and open source software.
If you encounter any problems, please file an issue along with a detailed description.
This project was generated from @cjolowicz's Hypermodern Python Cookiecutter template.