Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enhancement of chat template handling #4

Open
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

Nondzu
Copy link

@Nondzu Nondzu commented Feb 23, 2024

This PR introduces several enhancements. Specifically, it focuses on the wrap_chat_template and log_likelihood methods within the huggingface.py module.

Context of the Changes

These changes are based on the unmerged PR (EleutherAI#1287) from the original repository. The mentioned PR contains valuable improvements that are not yet merged into the main branch. By integrating these improvements into our fork, we aim to enhance the model's interaction with chat templates and accuracy in performance evaluation.

Changes Made

Instance Class Enhancements

  • args Property with Setter: A new property args with a setter has been added to the Instance class to update the instance's arguments. This ensures that the new arguments match the old ones in length and type, enhancing the integrity of instance modifications.

Huggingface Model Improvements

  • Chat Templating Settings: Added use_chat_template and system_prompt as optional parameters to the HFLM class to facilitate the handling of chat templates.
  • Template Wrapping Enhancement: The wrap_chat_template method has been improved to include system prompts and better prepare the context for text generation.
  • Generation Argument Refinement: Modified the handling of generation_kwargs to remove the temperature parameter when do_sample is set to False, providing a more consistent deterministic generation behavior.
  • Log Likelihood Updates: The loglikelihood method now utilizes the chat template functionality when enabled, with debug prints added for monitoring the prompt formatting process.
  • Rolling Log Likelihood Note: A note has been added to indicate that chat templates are currently not considered in per-token log likelihood evaluations, with a placeholder for a future warning message.

Additional Notes

  • Several debug print statements have been added to assist with further development and maintenance.

Impact

The proposed enhancements are expected to improve the model's flexibility and reliability when dealing with chat templates. They also aim to provide a more accurate and performant evaluation of the model's capabilities.

Please review the proposed changes and provide feedback on any further requirements or amendments needed.

I look forward to your feedback and am ready to make any necessary adjustments to prepare this PR for merging.

Thank you for considering this contribution to the project.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
1 participant