[Bug] Claude 3.5 在图片的内容分析上有问题

[breakstring]

📦 Environment

Vercel

📌 Version

1.1.6

💻 Operating System

macOS

🌐 Browser

Other

🐛 Bug Description

无法使用 Claude 3.5 解析图片内容，OpenAI 则没问题。

📷 Recurrence Steps

1. 选择模型为 Claude 3.5，插入图片到聊天窗口中，输入 prompt 请求解析。
2. 报错提示如下：
   、、、
   {
   "error": {
   "error": {
   "type": "error",
   "error": {
   "type": "invalid_request_error",
   "message": "messages.0.content.1.image.source.base64.media_type: Input should be 'image/jpeg', 'image/png', 'image/gif' or 'image/webp'"
   }
   },
   "status": 400,
   "headers": {
   "via": "1.1 google",
   "date": "Sun, 23 Jun 2024 23:28:03 GMT",
   "connection": "keep-alive",
   "request-id": "req_01BhNKq3EqhyioScf16uDPm5",
   "content-type": "application/json",
   "content-length": "193",
   "x-should-retry": "false",
   "x-cloud-trace-context": "b8d7344e29c2adb51dfd7c01eb5a9e9f"
   }
   },
   "endpoint": "https://api.anthropic.com",
   "provider": "anthropic"
   }
   、、、
   

🚦 Expected Behavior

2. 返回解析的内容

📝 Additional Information

No response

[lobehubbot]

👀 @breakstring

Thank you for raising an issue. We will investigate into the matter and get back to you as soon as possible.
Please make sure you have given us as much context as possible.
非常感谢您提交 issue。我们会尽快调查此事，并尽快回复您。 请确保您已经提供了尽可能多的背景信息。

[arvinxx]

看报错应该是 Anthropic 的api不支持base64 的图片，必须要可访问的 url 才能识别。你配置下s3应该就好了

[lobehubbot]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

The error message seems to be that Anthropic's API does not support base64 images, and an accessible URL is required to identify them. It should be fine if you configure s3

[breakstring]

看报错应该是 Anthropic 的api不支持base64 的图片，必须要可访问的 url 才能识别。你配置下s3应该就好了

我是采用的服务端数据库存储方案，目前是华为云兼容 S3 的 OBS。
BTW: 请求 OpenAI 正常。

[lobehubbot]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

Judging from the error report, it should be that Anthropic's API does not support base64 images, and an accessible URL is required to identify them. It should be fine if you configure s3

The server-side database storage solution I use is currently OBS compatible with Huawei Cloud's S3.
BTW: Requesting OpenAI works fine.

[arvinxx]

那估计是个 bug，我晚点看看

[lobehubbot]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

That's probably a bug, I'll check it out later

[breakstring]

那估计是个 bug，我晚点看看

Any update?

[arvinxx]

https://docs.anthropic.com/en/docs/build-with-claude/vision#can-claude-read-image-urls 查阅了下官方文档， Claude API 目前不支持传入 url 来识别，如果要使用必须要传输 base64。这么看和 Google 的 Gemini 问题一样，可能还是得用 base64 传输

[hominsu]

any update?

[huangyafei]

确实是个影响挺大的问题，请问有什么进展了么？

[lobehubbot]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

It is indeed an issue with a huge impact. Is there any progress?

[arvinxx]

Here is the PR to fix similar issue: #3370