This is an example Haystack 2.0 integration. It's an integration for Unstructured.io connectors. Please contribute 🚀
The current version has 2 available Unstructured connectors:
- Discord:
UnstructuredDiscordConnector - GitHub:
UnstructuredGitHubConnector - Google Drive:
UnstructuredGoogleDriveConnector
For example, you can write documents fetched from Discord using the UnstructuredDiscordConnector:
from haystack.preview import Pipeline
from haystack.preview.components.writers import DocumentWriter
from unstructured_haystack import UnstructuredDiscordConnector
from chroma_haystack import ChromaDocumentStore
# Chroma is used in-memory so we use the same instances in the two pipelines below
document_store = ChromaDocumentStore()
connector = UnstructuredDiscordConnector(api_key="UNSTRUCTURED_API_KEY", discord_token="DISCORD_TOKEN")
indexing = Pipeline()
indexing.add_component("connector", connector)
indexing.add_component("writer", DocumentWriter(document_store))
indexing.connect("connector.documents", "writer.documents")
indexing.run({"connector": {"channels" : "993539071815200889", "period": 3, "output_dir" : "discord-example"}})