[BUG MODEL]: Specifying a system prompt with tool_choice = auto leads to verbose answer instead of tools

[desaxce]

Input

Model

mistral-large-latest

Request Payload

messages = [
        {
            "role": "system",
            "content": "If needed, leverage the tools at your disposal to answer the user question, otherwise provide the answer."
        },
        {
            "role": "user", 
            "content": "What's the weather in Napoleon's hometown?"
        }
    ]

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_home_town",
            "description": "Get the home town of a specific person",
            "parameters": {
                "type": "object",
                "properties": {
                    "person": {
                        "type": "string",
                        "description": "The name of a person (first and last names) to identify.",
                    }
                },
                "required": ["person"],
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "get_current_weather",
            "description": "Get the current weather in a given location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {
                        "type": "string",
                        "description": "The city and state, e.g. San Francisco, CA",
                    },
                },
                "required": ["location"],
            },
        },
    },
]

settings = {
  "model": "mistral-large-latest",
  "temperature": 0,
  "tool_choice": "auto"
}

Output

{
  "role": "assistant",
  "content": "To answer your question, I need to first find out Napoleon's hometown and then check the current weather there. Let me use the functions provided to me.

First, I'll use the \"get_home_town\" function to find out Napoleon's hometown.

[{\"name\": \"get_home_town\", \"arguments\": {\"person\": \"Napoleon Bonaparte\"}}]"
}

Expected Behavior

The user question is "What's the weather in Napoleon's hometown?".

I expected the model to choose the first tool get_home_town() with a Napoleon parameter.

Instead, the model decided to answer a plain-text in english describing the tool call to make. Although the described tool call is the right one, I expect to get an actual tool call as output.

I expected the following:

[
  {
    "id": "lT7H41Y0R",
    "type": "function",
    "function": {
      "name": "get_home_town",
      "arguments": "{\"person\": \"Napoleon\"}"
    }
  }
]

Interestingly, I do get mistral-large-latest to yield this tool call if I remove the system prompt.

Additional Context

Here are the traces for my trials:

• Verbose answer (no tool call): Wrong behavior
• Expected behavior (got it without system message): Right behavior

Suggested Solutions

For now, my solution is to not specify a system message to avoid the model going verbose.

Inherent problem seems to be at train level, where the model seems to have seen less system + user messages followed with a <|startoftool|> token. Speculating here though.

[fuegoio]

Thanks for opening this issue. We are aware that our models might have issues with function calling. What you said with system prompts is really interesting, I am escalating this to our science team.

The next generation of our models will be much better :) Thanks for the feedback anyway, I'll close as there is nothing we can do at the moment on the client side.