Marquez is an open source metadata service for the collection, aggregation, and visualization of a data ecosystem's metadata. It maintains the provenance of how datasets are consumed and produced, provides global visibility into job runtime and frequency of dataset access, centralization of dataset lifecycle management, and much more.

This project is under active development at The We Company and Stitch Fix (in collaboration with many others organizations).

We invite everyone to help us improve and keep documentation up to date. Documentation is maintained in this repository and can be found under docs/.

• Java 8+
• PostgreSQL 9.6
• Gradle 4.10+ (optional)
• Docker 17.05+ (optional)

To build the entire project run:

$ ./gradlew shadowJar

The executable can be found under build/libs/

To run Marquez, you will have to define config.yml. The configuration file is passed to the application and used to specify your database connection. When creating your database using createdb, we recommend calling it marquez:

$ createdb marquez;

With your database created, you can now copy config.example.yml:

$ cp config.example.yml config.yml

You will then need to set the following environment variables (we recommend adding them to your .bashrc): POSTGRES_DB, POSTGRES_USER, and POSTGRES_PASSWORD. The environment variables override the equivalent option in the configuration file.

By default, Marquez uses the following ports:

• TCP port 8080 is available for the HTTP API server.
• TCP port 8081 is available for the admin interface.

Note: All of the configuration settings in config.yml can be specified either in the configuration file or in an environment variable.

$ ./gradlew run --args 'server config.yml'

Then browse to the admin interface: http:https://localhost:8081

$ docker-compose up

Marquez listens on port 5000 for all API calls and port 5001 for the admin interface.

• Website: https://marquezproject.github.io/marquez
• Mailing Lists:
  • [email protected] (user support and questions)
  • [email protected] (development discussions)
• Chat: https://gitter.im/marquez-project/community
• Twitter: @MarquezProject

See CONTRIBUTING.md for more details about how to contribute.