This repository contains the implementation of FigurA11y, a system presented in our IUI 2024 paper. FigurA11y assists authors in creating high-quality alt text for scientific figures using a human-AI collaborative approach.
This code is primarily intended for research purposes and may not be suitable for production deployments.
FigurA11y combines a Flask-based backend and a NextJS frontend. The backend handles PDF processing and metadata extraction, while the frontend provides an interactive interface for alt text editing.
Before following the steps below, make sure that you have the latest version of Docker 🐳 installed on your local machine.
For AI suggestions, you will also need an OpenAI API key. Once you have one, place it in ui/.env:
OPENAI_API_KEY="XXXXXXXXXXXXXXXX"You will also need to download the DocFigure pretrained checkpoint as api/models/docfigure.pth.
To start a version of the application locally for development purposes after cloning the repository, run:
~ docker compose up --buildBy default, the following command makes a development build:
docker-compose up --buildThis enables features like live updates, debug logs, etc.
For a production build, instead use:
docker-compose -f docker-compose.prod.yaml up --build/app: Main application code./models: Machine learning model wrappers for pipeline preprocessing steps./services: Services defining preprocessing steps to be composed into the pipeline./api: Definition of API endpoints. Structured as a collection of blueprints.
/demo: Other logging dependencies.
/components: React components for UI building blocks./description-editor: Components for description editor.
/context: React context for global state management (currently used for settings)./hooks: Custom React hooks for shared logic (mainly used for the Zustand stores)/lib: Utility functions and classes./pages: NextJS pages./api: API routes (used for suggestions, via OpenAI API).