Python client use async with stream not work!

[iyuhang]

I use python client

use the code from document , use stream and async

It raise : httpx.ResponseNotRead: Attempted to access streaming response content, without having called read().

from mistralai.async_client import MistralAsyncClient
from mistralai.models.chat_completion import ChatMessage

api_key = os.environ["MISTRAL_API_KEY"]
model = "mistral-large-latest"

client = MistralAsyncClient(api_key=api_key)

messages = [
    ChatMessage(role="user", content="What is the best French cheese?")
]

# With async
async_response = client.chat_stream(model=model, messages=messages)

async for chunk in async_response: 
    print(chunk.choices[0].delta.content)

httpx.ResponseNotRead: Attempted to access streaming response content, without having called `read()`.
2024-02-29 15:13:08,972 ERROR sanic.error: Exception occurred while handling uri: 'http:https://localhost:9000/message/doris'
Traceback (most recent call last):
  File "/Users/apple/opt/anaconda3/envs/rena-sanic-39/lib/python3.9/site-packages/sanic/app.py", line 1385, in handle_request
    response = await response
  File "/Users/apple/SynologyDrive/code/rena_sanic/srf/views.py", line 71, in view
    return await self.dispatch(request, *args, **kwargs)
  File "/Users/apple/SynologyDrive/code/rena_sanic/srf/views.py", line 109, in dispatch
    response = await run_awaitable(handler, request=request, *args, **kwargs)
  File "/Users/apple/SynologyDrive/code/rena_sanic/srf/utils.py", line 95, in run_awaitable
    return await func(*args, **kwargs) if inspect.iscoroutinefunction(func) else func(*args, **kwargs)
  File "/Users/apple/SynologyDrive/code/rena_sanic/message/views.py", line 102, in post
    await ChatManager().ask_stream(conversation_id=conversation_id, response=responses, content=message,
  File "/Users/apple/SynologyDrive/code/rena_sanic/mages/logic/chat_manager.py", line 36, in ask_stream
    async for data in MistralComplete().complete_stream(model="mistral-large-latest", messages=content, temperature=0.7):
  File "/Users/apple/SynologyDrive/code/rena_sanic/mages/logic/completes/mistral.py", line 23, in complete_stream
    async for chunk in async_response:
  File "/Users/apple/opt/anaconda3/envs/rena-sanic-39/lib/python3.9/site-packages/mistralai/async_client.py", line 223, in chat_stream
    async for json_response in async_response:
  File "/Users/apple/opt/anaconda3/envs/rena-sanic-39/lib/python3.9/site-packages/mistralai/async_client.py", line 85, in _request
    self._check_streaming_response(response)
  File "/Users/apple/opt/anaconda3/envs/rena-sanic-39/lib/python3.9/site-packages/mistralai/client_base.py", line 145, in _check_streaming_response
    self._check_response_status_codes(response)
  File "/Users/apple/opt/anaconda3/envs/rena-sanic-39/lib/python3.9/site-packages/mistralai/client_base.py", line 137, in _check_response_status_codes
    message=f"Status: {response.status_code}. Message: {response.text}",
  File "/Users/apple/opt/anaconda3/envs/rena-sanic-39/lib/python3.9/site-packages/httpx/_models.py", line 573, in text
    content = self.content
  File "/Users/apple/opt/anaconda3/envs/rena-sanic-39/lib/python3.9/site-packages/httpx/_models.py", line 567, in content
    raise ResponseNotRead()
httpx.ResponseNotRead: Attempted to access streaming response content, without having called `read()`.

[sebampuero]

Since you are trying to call the async client, you need to call it inside an async loo. Try encapsulating the call to async_response = client.chat_stream(model=model, messages=messages) and the for loop after inside an async function, then call it using asyncio.run(my_async_func())

[drpebcak]

Hmm.. I am getting this same error when using the async client to do function calling. Works just fine up until a function response is sent.. then throws this httpx.ResponseNotRead error. Specifically this happens when I'm trying to iterate over the response using an async for ... loop.

I am running inside an async function.

[aanaseer]

Which version of mistralai are you using? (pip show mistralai)

I am able to run the following code snippet below with no errors on v0.0.11 through to v0.1.6.

import asyncio
import os

from mistralai.async_client import MistralAsyncClient
from mistralai.models.chat_completion import ChatMessage

api_key = os.environ["MISTRAL_API_KEY"]
model = "mistral-large-latest"

client = MistralAsyncClient(api_key=api_key)

messages = [ChatMessage(role="user", content="What is the best French cheese?")]

# With async
async_response = client.chat_stream(model=model, messages=messages)


async def main():
  async for chunk in async_response:
      print(chunk.choices[0].delta.content)


# Run the async function
asyncio.run(main())

[drpebcak]

I was using latest, but it turned out that there was an error in my message syntax that was being bubbled up this way. The error message I received was a bit of a red herring.