Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change last_n_messages to default to 'auto' #1356

Merged
merged 5 commits into from
Jan 23, 2024

Conversation

afourney
Copy link
Member

Why are these changes needed?

An 'auto' mode was added to the last_n_messages code execution configuration in #693. Since then it has been my top recommended setting for people struggling with GroupChat. Basically, if last_n_messages is set too low (e.g., the current default of 1), then code execution can break if another agent speaks between when code is written and when it is set to be executed. 'Auto' avoid this problem.

In non-GroupChat settings, last_n_message = 'auto' and last_n_message = 1 work identically. There would be no change in behavior.

TL;DR;
Though 'auto' can vastly improve GroupChat performance (e.g., see my experiments with HumanEval), it is still not the default. This PR changes that.

Related issue number

#693

Checks

@codecov-commenter
Copy link

codecov-commenter commented Jan 21, 2024

Codecov Report

Attention: 3 lines in your changes are missing coverage. Please review.

Comparison is base (05923a7) 32.45% compared to head (214012e) 14.18%.

Files Patch % Lines
autogen/agentchat/conversable_agent.py 0.00% 3 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##             main    #1356       +/-   ##
===========================================
- Coverage   32.45%   14.18%   -18.28%     
===========================================
  Files          41       39        -2     
  Lines        4905     4899        -6     
  Branches     1119     1120        +1     
===========================================
- Hits         1592      695      -897     
- Misses       3187     4178      +991     
+ Partials      126       26      -100     
Flag Coverage Δ
unittests 14.18% <0.00%> (-18.23%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@afourney
Copy link
Member Author

afourney commented Jan 22, 2024

Just to add some data to support this move.

I used AutoGenBench to run a random 10% subsample of HumanEval through the GroupChatThreeAgents_Distractor scenario in two conditions: last_n_messages = 1 (the current default), and last_n_messages = "auto". I used GPT-3.5-turbo-16k. The results are as follows:

                 Trial 0             Trial 1
		 last_n = 1    last_n = auto
Task Id          Success             Success
-------------  ---------           ---------
HumanEval_107      False                True
HumanEval_22       False                True
HumanEval_43        True                True
HumanEval_88        True                True
HumanEval_14       False                True
HumanEval_157      False                True
HumanEval_141      False                True
HumanEval_57       False                True
HumanEval_154       True               False
HumanEval_153      False                True
HumanEval_93       False                True
HumanEval_137      False               False
HumanEval_143       True                True
HumanEval_13        True                True
HumanEval_49        True                True
HumanEval_95       False                True
-------------  ---------           ---------
Successes              6                  14
Failures              10                   2
Missing                0                   0
Total                 16                  16 

@sonichi sonichi added this pull request to the merge queue Jan 23, 2024
Merged via the queue into main with commit c05e212 Jan 23, 2024
103 of 113 checks passed
@afourney afourney deleted the change_last_n_messages_default branch January 23, 2024 17:13
mtwalther pushed a commit to mtwalther/autogen that referenced this pull request Jan 26, 2024
* Change last_n_messages to default to 'auto'

* Added last_n_messages validation, as per Eric's comment.

---------

Co-authored-by: Davor Runje <[email protected]>
corleroux pushed a commit to corleroux/autogen that referenced this pull request Jan 30, 2024
* Change last_n_messages to default to 'auto'

* Added last_n_messages validation, as per Eric's comment.

---------

Co-authored-by: Davor Runje <[email protected]>
@sonichi sonichi mentioned this pull request Feb 2, 2024
3 tasks
whiskyboy pushed a commit to whiskyboy/autogen that referenced this pull request Apr 17, 2024
* Change last_n_messages to default to 'auto'

* Added last_n_messages validation, as per Eric's comment.

---------

Co-authored-by: Davor Runje <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants