List of AI tools that can interact with user interfaces

• ScreenAI
• Llava
• SegmentEverythingEverywhereAllAtOnce

These are still mostly text-based

• AgentLLM
• LaVague

• HyperWrite AI Agent

• CogAgent: CogAgent is an open-source visual language model that can identify regions of areas of UIs to interact with.
• AIOS: Can interact with operating system

• OpenAdapt.AI: AI-First Process Automation with Large ([Language (LLMs) / Action (LAMs) / Multimodal (LMMs)] / Visual Language (VLMs)) Models
• ScreenAgent
• Mobile-Agent
• UI-ACT: An AI agent for interacting with a computer using the graphical user interface
• OpenInterpreter: Uses code to interact with operating system.

• Adept: Company looking to automate everything in software

• Autonomous Interactive Agents: MIT
• Toolformer
• Visual Programming: Compositional visual reasoning without training