txtai executes machine-learning workflows to transform data and build AI-powered semantic search applications.

Traditional search systems use keywords to find data. Semantic search applications have an understanding of natural language and identify results that have the same meaning, not necessarily the same keywords.

Backed by state-of-the-art machine learning models, data is transformed into vector representations for search (also known as embeddings). Innovation is happening at a rapid pace, models can understand concepts in documents, audio, images and more.

Summary of txtai features:

• 🔎 Large-scale similarity search with multiple index backends (Faiss, Annoy, Hnswlib)
• 📄 Create embeddings for text snippets, documents, audio, images and video. Supports transformers and word vectors.
• 💡 Machine-learning pipelines to run extractive question-answering, zero-shot labeling, transcription, translation, summarization and text extraction
• ↪️️ Workflows that join pipelines together to aggregate business logic. txtai processes can be microservices or full-fledged indexing workflows.
• 🔗 API bindings for JavaScript, Java, Rust and Go
• ☁️ Cloud-native architecture that scales out with container orchestration systems (e.g. Kubernetes)

Applications range from similarity search to complex NLP-driven data extractions to generate structured databases. The following applications are powered by txtai.

txtai is built with Python 3.7+, Hugging Face Transformers, Sentence Transformers and FastAPI

In addition to traditional search systems, a growing number of semantic search solutions are available, so why txtai?

• pip install txtai is all you need

# Get started in a couple lines
from txtai.embeddings import Embeddings

embeddings = Embeddings({"path": "sentence-transformers/all-MiniLM-L6-v2"})
embeddings.index([(0, "Correct", None), (1, "Not what we hoped", None)])
embeddings.search("positive", 1)
#[(0, 0.2986203730106354)]

• Works well with both small and big data - scale up as needed
• Rich data processing framework (pipelines and workflows) to pre and post process data
• Work in your programming language of choice via the API
• Modular with low footprint - install additional dependencies when you need them
• Learn by example - notebooks cover all available functionality

The easiest way to install is via pip and PyPI

pip install txtai

Python 3.7+ is supported. Using a Python virtual environment is recommended.

See the detailed install instructions for more information covering installing from source, environment specific prerequisites and optional dependencies.

The examples directory has a series of notebooks and applications giving an overview of txtai. See the sections below.

Build semantic/similarity/vector search applications.

Notebook	Description
Introducing txtai	Overview of the functionality provided by txtai
Build an Embeddings index with Hugging Face Datasets	Index and search Hugging Face Datasets
Build an Embeddings index from a data source	Index and search a data source with word embeddings
Add semantic search to Elasticsearch	Add semantic search to existing search systems
API Gallery	Using txtai in JavaScript, Java, Rust and Go
Similarity search with images	Embed images and text into the same space for search
Distributed embeddings cluster	Distribute an embeddings index across multiple data nodes
What's new in txtai 4.0	Content storage, SQL, object storage, reindex and compressed indexes

Transform data with NLP-backed pipelines.

Notebook	Description
Extractive QA with txtai	Introduction to extractive question-answering with txtai
Extractive QA with Elasticsearch	Run extractive question-answering queries with Elasticsearch
Extractive QA to build structured data	Build structured datasets using extractive question-answering
Apply labels with zero shot classification	Use zero shot learning for labeling, classification and topic modeling
Building abstractive text summaries	Run abstractive text summarization
Extract text from documents	Extract text from PDF, Office, HTML and more
Transcribe audio to text	Convert audio files to text
Translate text between languages	Streamline machine translation and language detection

Efficiently process data at scale.

Notebook	Description
Run pipeline workflows	Simple yet powerful constructs to efficiently process data
Transform tabular data with composable workflows	Transform, index and search tabular data
Tensor workflows	Performant processing of large tensor arrays

Train NLP models.

Notebook	Description
Train a text labeler	Build text sequence classification models
Train without labels	Use zero-shot classifiers to train new models
Train a QA model	Build and fine-tune question-answering models
Export and run models with ONNX	Export models with ONNX, run natively in JavaScript, Java and Rust
Export and run other machine learning models	Export and run models from scikit-learn, PyTorch and more

Series of example applications with txtai. Links to hosted versions on Hugging Face Spaces also provided.

Full documentation on txtai including configuration settings for pipelines, workflows, indexing and the API.

• Introducing txtai, AI-powered semantic search built on Transformers
• Run machine-learning workflows to transform data and build AI-powered semantic search applications with txtai
• Semantic search on the cheap
• Tutorial series on dev.to

For those who would like to contribute to txtai, please see this guide.