This fork of autogen-magentic-one enables compatibility with an Ollama server running the Llama-3.2-11B-Vision model. It allows the magentic-one agent workflow to operate fully locally, removing the need for an OpenAI API key. While the results may not fully match the quality returned by GPT-4, this setup offers significant cost savings and provides a solid foundation for expanding local agent capabilities in magentic-one.

For setup, please follow the original installation instructions from the Microsoft autogen-magentic-one repository. This fork retains the core structure of the original repository, with changes limited to specific scripts to enable the use of the Ollama server and model in place of the OpenAI API.

This fork primarily modifies the following scripts within the magentic-one component of the autogen library:

1. examples/example.py: Changed start page to Google.

2. examples/example_websurfer.py: Changed start page to Google.

3. interface/magentic_one_helper.py: Updated helper functions for compatibility with Ollama server.

4. src/autogen_magentic_one/agents/multimodal_web_surfer/multimodal_web_surfer.py:

   • Enforced structured JSON responses for web actions, including:
     • Setting json_output=True in model client calls.
     • Updating text prompts to require single JSON tool calls.
     • Adding clear examples for each web interaction type.
     • Removing narrative responses from prompt templates.
   • Improved parser to handle pure JSON responses consistently, ensuring reliable, structured outputs from the LLM.

5. src/autogen_magentic_one/utils.py: Major changes to replace OpenAI/Azure support with Ollama-specific implementation.

   Major Changes

   • Replaced OpenAI/Azure implementation with a custom Ollama-specific chat completion client.
   • Added support for base64 image encoding for vision models.
   • Integrated a new Ollama API client for handling chat completions.

   Detailed Changes

   • New Features:

     • Introduced OllamaConfig dataclass with Ollama-specific configuration options:
       • base_url (default: https://localhost:11434)
       • model (default: "llama3.2-vision")
       • Parameters for temperature and top_p tuning.

   • Modified Components:

     • Implemented the OllamaChatCompletionClient class with:
       • Vision model support and base64 image encoding.
       • Asynchronous API communication with Ollama server.
       • Custom message processing for multimodal inputs.

   • API Changes:

     • Updated the create() method to handle:
       • Image-text multimodal inputs.
       • Ollama’s specific API format.
       • Streaming responses with enhanced error handling for server communication.

   • Removed Features:

     • Removed Azure AD token provider functionality.
     • Removed OpenAI-specific provider selection logic.
     • Removed all Azure/OpenAI client implementations.

   • Dependencies:

     • Added dependencies:
       • aiohttp for asynchronous HTTP communication.
       • base64 for image encoding support.
     • Removed dependencies:
       • Azure identity and OpenAI client libraries.

   • Breaking Changes:

     • Environment variables are now configured for Ollama-specific parameters.
     • Provider selection logic removed in favor of direct Ollama implementation.
     • Authentication mechanism switched to Ollama’s local server setup, removing Azure/OpenAI dependencies.

This section provides an overview of the changes made specifically to support the Ollama server setup within magentic-one. For general information on autogen, please refer to the original README below.

AutoGen is an open-source framework for building AI agent systems. It simplifies the creation of event-driven, distributed, scalable, and resilient agentic applications. It allows you to quickly build systems where AI agents collaborate and perform tasks autonomously or with human oversight.

• Key Features
• API Layering
• Quickstart
• Roadmap
• FAQs

AutoGen streamlines AI development and research, enabling the use of multiple large language models (LLMs), integrated tools, and advanced multi-agent design patterns. You can develop and test your agent systems locally, then deploy to a distributed cloud environment as your needs grow.

AutoGen offers the following key features:

• Asynchronous Messaging: Agents communicate via asynchronous messages, supporting both event-driven and request/response interaction patterns.
• Full type support: use types in all interfaces and enforced type check on build, with a focus on quality and cohesiveness
• Scalable & Distributed: Design complex, distributed agent networks that can operate across organizational boundaries.
• Modular & Extensible: Customize your system with pluggable components: custom agents, tools, memory, and models.
• Cross-Language Support: Interoperate agents across different programming languages. Currently supports Python and .NET, with more languages coming soon.
• Observability & Debugging: Built-in features and tools for tracking, tracing, and debugging agent interactions and workflows, including support for industry standard observability with OpenTelemetry

↑ Back to Top ↑

AutoGen has several packages and is built upon a layered architecture. Currently, there are three main APIs your application can target:

• Core
• AgentChat
• Extensions

• Installation
• Quickstart

The core API of AutoGen, autogen-core, is built following the actor model. It supports asynchronous message passing between agents and event-based workflows. Agents in the core layer handle and produce typed messages, using either direct messaging, which functions like RPC, or via broadcasting to topics, which is pub-sub. Agents can be distributed and implemented in different programming languages, while still communicating with one another. Start here if you are building scalable, event-driven agentic systems.

• Installation
• Quickstart

The AgentChat API, autogen-agentchat, is task driven and at a high level like AutoGen 0.2. It allows you to define conversational agents, compose them into teams and then use them to solve tasks. AgentChat itself is built on the core layer, but it abstracts away much of its low-level system concepts. If your workflows don't fit into the AgentChat API, target core instead. Start here if you just want to focus on quickly getting started with multi-agents workflows.

The extension package autogen-ext contains implementations of the core interfaces using 3rd party systems, such as OpenAI model client and Azure code executors. Besides the built-in extensions, the package accommodates community-contributed extensions through namespace sub-packages. We look forward to your contributions!

↑ Back to Top ↑

First install the packages:

pip install 'autogen-agentchat==0.4.0.dev6' 'autogen-ext[openai]==0.4.0.dev6'

The following code uses OpenAI's GPT-4o model and you need to provide your API key to run. To use Azure OpenAI models, follow the instruction here.

import asyncio
from autogen_agentchat.agents import AssistantAgent
from autogen_agentchat.task import Console, TextMentionTermination
from autogen_agentchat.teams import RoundRobinGroupChat
from autogen_ext.models import OpenAIChatCompletionClient

# Define a tool
async def get_weather(city: str) -> str:
    return f"The weather in {city} is 73 degrees and Sunny."

async def main() -> None:
    # Define an agent
    weather_agent = AssistantAgent(
        name="weather_agent",
        model_client=OpenAIChatCompletionClient(
            model="gpt-4o-2024-08-06",
            # api_key="YOUR_API_KEY",
        ),
        tools=[get_weather],
    )

    # Define termination condition
    termination = TextMentionTermination("TERMINATE")

    # Define a team
    agent_team = RoundRobinGroupChat([weather_agent], termination_condition=termination)

    # Run the team and stream messages to the console
    stream = agent_team.run_stream(task="What is the weather in New York?")
    await Console(stream)

asyncio.run(main())

The .NET SDK does not yet support all of the interfaces that the python SDK offers but we are working on bringing them to parity. To use the .NET SDK, you need to add a package reference to the src in your project. We will release nuget packages soon and will update these instructions when that happens.

git clone https://github.com/microsoft/autogen.git
cd autogen
# Switch to the branch that has this code
git switch staging-dev
# Build the project
cd dotnet && dotnet build AutoGen.sln
# In your source code, add AutoGen to your project
dotnet add <your.csproj> reference <path to your checkout of autogen>/dotnet/src/Microsoft.AutoGen/Agents/Microsoft.AutoGen.Agents.csproj

Then, define and run your first agent:

using Microsoft.AutoGen.Abstractions;
using Microsoft.AutoGen.Agents;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;

// send a message to the agent
var app = await App.PublishMessageAsync("HelloAgents", new NewMessageReceived
{
    Message = "World"
}, local: true);

await App.RuntimeApp!.WaitForShutdownAsync();
await app.WaitForShutdownAsync();

[TopicSubscription("HelloAgents")]
public class HelloAgent(
    IAgentContext context,
    [FromKeyedServices("EventTypes")] EventTypes typeRegistry) : ConsoleAgent(
        context,
        typeRegistry),
        ISayHello,
        IHandle<NewMessageReceived>,
        IHandle<ConversationClosed>
{
    public async Task Handle(NewMessageReceived item)
    {
        var response = await SayHello(item.Message).ConfigureAwait(false);
        var evt = new Output
        {
            Message = response
        }.ToCloudEvent(this.AgentId.Key);
        await PublishEvent(evt).ConfigureAwait(false);
        var goodbye = new ConversationClosed
        {
            UserId = this.AgentId.Key,
            UserMessage = "Goodbye"
        }.ToCloudEvent(this.AgentId.Key);
        await PublishEvent(goodbye).ConfigureAwait(false);
    }
    public async Task Handle(ConversationClosed item)
    {
        var goodbye = $"*********************  {item.UserId} said {item.UserMessage}  ************************";
        var evt = new Output
        {
            Message = goodbye
        }.ToCloudEvent(this.AgentId.Key);
        await PublishEvent(evt).ConfigureAwait(false);
        await Task.Delay(60000);
        await App.ShutdownAsync();
    }
    public async Task<string> SayHello(string ask)
    {
        var response = $"\n\n\n\n***************Hello {ask}**********************\n\n\n\n";
        return response;
    }
}
public interface ISayHello
{
    public Task<string> SayHello(string ask);
}

dotnet run

↑ Back to Top ↑

• AutoGen 0.2 - This is the current stable release of AutoGen. We will continue to accept bug fixes and minor enhancements to this version.
• AutoGen 0.4 - This is the first release of the new architecture. This release is still in preview. We will be focusing on the stability of the interfaces, documentation, tutorials, samples, and a collection of built-in agents which you can use. We are excited to work with our community to define the future of AutoGen. We are looking for feedback and contributions to help shape the future of this project. Here are some major planned items:
  • More programming languages (e.g., TypeScript)
  • More built-in agents and multi-agent workflows
  • Deployment of distributed agents
  • Re-implementation/migration of AutoGen Studio
  • Integration with other agent frameworks and data sources
  • Advanced RAG techniques and memory services

↑ Back to Top ↑

AutoGen v0.4 is a rewrite of AutoGen from the ground up to create a more robust, scalable, easier to use, cross-language library for building AI Agents. Some key features include asynchronous messaging, support for scalable distributed agents, modular extensible design (bring your own agents, implement behaviors however you like), cross-language support, improved observability, and full typing integration. It is a breaking change.

We listened to our AutoGen users, learned from what was working, and adapted to fix what wasn't. We brought together wide-ranging teams working on many different types of AI Agents and collaborated to design an improved framework with a more flexible programming model and better scalability.

We want to reaffirm our commitment to supporting both the original version of AutoGen (0.2) and the redesign (0.4) . AutoGen 0.4 is still work-in-progress, and we shared the code now to build with the community. There are no plans to deprecate the original AutoGen anytime soon, and both versions will be actively maintained.

This code is still experimental, so expect changes and bugs while we work towards a stable 0.4 release. We encourage early adopters to try it out, give us feedback, and contribute. For those looking for a stable version we recommend to continue using 0.2

If you consider yourself an early adopter, you are comfortable making some changes to your code, and are willing to try it out, then yes.

AutoGen 0.2 can be installed with:

pip install autogen-agentchat~=0.2

Yes, this is on the roadmap. Our current plan is to enable an implementation of AutoGen Studio on the AgentChat high level API which implements a set of agent functionalities (agents, teams, etc).

For users familiar with AutoGen, the AgentChat library in 0.4 provides similar concepts. We are working on a migration guide.

We are still actively developing AutoGen 0.4. One exciting new feature is the emergence of new SDKs for .NET. The python SDKs are further ahead at this time but our goal is to achieve parity. We aim to add additional languages in future releases.

We are still working on improving the documentation, samples, and enhancing the code. We are hoping to release before the end of the year when things are ready.

The rearchitecture of the framework started with multiple Microsoft teams coming together to address the gaps and learnings from AutoGen 0.2 - merging ideas from several predecessor projects. The team worked on this internally for some time to ensure alignment before moving work back to the open in October 2024.

Use GitHub Issues for bug reports and feature requests. Use GitHub Discussions for general questions and discussions.

We are unable to use Discord for project discussions. Therefore, we request that all discussions take place on https://github.com/microsoft/autogen/discussions/ going forward.

https://github.com/microsoft/autogen/ remains the only official repo for development and support of AutoGen. We are aware that there are thousands of forks of AutoGen, including many for personal development and startups building with or on top of the library. We are not involved with any of these forks and are not aware of any plans related to them.

Our project remains fully open-source and accessible to everyone. We understand that some forks use different licenses to align with different interests. We will continue to use the most permissive license (MIT) for the project.

Currently, we are unable to make releases to the pyautogen package via Pypi due to a change to package ownership that was done without our involvement. Additionally, we are moving to using multiple packages to align with the new design. Please see details here.

We are grateful to all the contributors to AutoGen 0.2 and we look forward to continuing to collaborate with everyone in the AutoGen community.

↑ Back to Top ↑

Microsoft and any contributors grant you a license to the Microsoft documentation and other content in this repository under the Creative Commons Attribution 4.0 International Public License, see the LICENSE file, and grant you a license to any code in the repository under the MIT License, see the LICENSE-CODE file.

Microsoft, Windows, Microsoft Azure, and/or other Microsoft products and services referenced in the documentation may be either trademarks or registered trademarks of Microsoft in the United States and/or other countries. The licenses for this project do not grant you rights to use any Microsoft names, logos, or trademarks. Microsoft's general trademark guidelines can be found at https://go.microsoft.com/fwlink/?LinkID=254653.

Privacy information can be found at https://go.microsoft.com/fwlink/?LinkId=521839

Microsoft and any contributors reserve all other rights, whether under their respective copyrights, patents, or trademarks, whether by implication, estoppel, or otherwise.

↑ Back to Top ↑