-
Notifications
You must be signed in to change notification settings - Fork 1.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Research and test different techniques to improve performance on table QA #7870
Comments
I think in addition to the proposed points (which mostly focus on retrieval), it would also be good to add tasks that focus on answer generation with LLMs. For example,
Also for the retrieval side of things:
|
@sjrl In my experience, basic arithmetic has been a challenge for all LLMs I've worked with. Even though some LLMs, like code models, might perform slightly better at these tasks, their accuracy remains inconsistent, making them unreliable for production use. For indexing PDFs, I suggest we could develop an agent that processes the document chunk by chunk in an interactive manner, extracting specific facts such as a company's net profit. This agent could add references to tables, images, and other relevant elements, which we could then use to enhance the metadata of those elements. By indexing all chunks in a document store and maintaining separate storage for tables and images, we could preserve context during retrieval. When a table needs to be accessed, the retrieval process would pull up a passage that references it. Additionally, filtering rows and columns of tables before passing them to the LLM might help avoid confusion. I understand this needs testing, but I am confident in its potential. I'll start working on this after my exams. |
These are some techniques to assess and decide whether it's worth trying or not, but it's up to the assignee to try different techniques that will work on the selected dataset.
The text was updated successfully, but these errors were encountered: