Support including file attachments in the chat message #957
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Now that models have much larger context windows, we can reasonably include full texts of certain files in the messages. Do this when an explicit file filter is set in a conversation. Do so in a separate user message in order to mitigate any confusion in the operation. Pipe the relevant attached_files context through all methods calling into models. We'll want to limit the file sizes for which this is used and provide more helpful UI indicators that this sort of behavior is taking place.
…e-full-file-in-convo-with-filter
…raw text, before further processing
- weave through all subsequent subcalls to models, where relevant, and save to conversation log
- Document is first converted in the chatinputarea, then sent to the chat component. From there, it's sent in the chat API body and then processed by the backend - We couldn't directly use a UploadFile type in the backend API because we'd have to convert the api type to a multipart form. This would require other client side migrations without uniform benefit, which is why we do it in this two-phase process. This also gives us capacity to repurpose the moe generic interface down the road.
…rsation history - When chatting on a shared page, fork and redirect to a new conversation page
…e-full-file-in-convo-with-filter
debanjum
reviewed
Nov 10, 2024
…e structured messages
…e-full-file-in-convo-with-filter
Keep function where it original was allows tracking diffs and change history more easily
Use python standard method tempfile.NamedTemporaryFile to write, delete temporary files safely.
…b app Remove unused total size calculations in chat input
debanjum
approved these changes
Nov 11, 2024
| @@ -149,6 +155,7 @@ def converse_anthropic( | |||
| query_images: Optional[list[str]] = None, | |||
| vision_available: bool = False, | |||
| tracer: dict = {}, | |||
| attached_files: str = None, | |||
There was a problem hiding this comment.
I was using query_{x} pattern to indicate it's part of the user query (as opposed to files or images retrieved by Khoj). To me attached files doesn't make it clear where it got attached or who attached the file 😅 . Not sure what would be a better name for the query_images, attached_files args.
But it'd be nice if they have similar names and have attached_files and query_images args next to each other as they have similar origins and expected handling, IMO. But not a blocker
…hub.com:khoj-ai/khoj into features/include-full-file-in-convo-with-filter
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Now that models have much larger context windows, we can reasonably include full texts of certain files in the messages. Do this when an explicit file filter is set in a conversation. Do so in a separate user message in order to mitigate any confusion in the operation.
Pipe the relevant attached_files context through all methods calling into models.
This breaks certain prior behaviors. We will no longer automatically be processing/generating embeddings on the backend and adding documents to the "brain". You'll have to go to settings and go through the upload documents flow there in order to add docs to the brain (i.e., have search include them during question / response).