fix: Set model_max_length in the Tokenizer of DefaultPromptHandler

[bogdankostic]

Related Issues

• fixes "Token indices sequence length is longer than the specified maximum sequence length for this model" with Cohere Command #5589

Proposed Changes:

This PR sets the model_max_length parameter of the tokenizer of DefaultPromptHandler.

How did you test it?

I added a unit test + added a test case in an integration test.

Notes for the reviewer

Without this change, users are getting a warning message from the transformers library that the sequence length is too long for the model they are using in case they are using models supporting larger sequence length than 1024.

Checklist

• I have read the contributors guidelines and the code of conduct
• I have updated the related issue with new insights and changes
• I added unit tests and updated the docstrings
• I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test:.
• I documented my code
• I ran pre-commit hooks and fixed any issue

[coveralls]

Pull Request Test Coverage Report for Build 5903109928

• 0 of 0 changed or added relevant lines in 0 files are covered.
• 1 unchanged line in 1 file lost coverage.
• Overall coverage decreased (-0.002%) to 48.257%

---

Files with Coverage Reduction	New Missed Lines	%
utils/context_matching.py	1	95.7%

Totals
Change from base Build 5901803206:	-0.002%
Covered Lines:	11460
Relevant Lines:	23748

---

💛 - Coveralls