🔥This work aims to explore the debating capability of LLMs by proposing the MAD framework, which stands for Multi-Agents Debate.

"Truth emerges from the clash of adverse ideas."
"真理越辩越明。"

The cognitive behavior of large language models (LLMs) has garnered significant attention in recent times. For example, self-reflection, a concept that usually refers to the process of introspection and examination of a person's own thoughts, has also been demonstrated effective with LLMs in solving challenging NLP tasks. However, we point out that self-reflection can easily fall into the degeneration of thoughts (DoT) issue in the follow scenarios:

• Bias and Distorted Perception: Self-perception can be influenced by biases, preconceived notions, and distorted thinking patterns. If an individual's self-reflection is clouded by such biases or distorted thinking, it can lead to 😔inaccurate conclusions and hinder personal growth.
• Rigidity and Resistance to Change: Self-reflection often involves challenging one's beliefs, assumptions, and behaviors. If an individual is resistant to change or holds rigid beliefs, they may 😔struggle to engage in meaningful self-reflection that leads to personal growth.
• Limited External Feedback: Self-reflection is primarily an internal process, but external feedback can provide valuable perspectives and insights. Without seeking or considering external feedback, an individual may 😔miss important blind spots or alternative viewpoints that can enrich their self-reflection.

Figure 1: Comparison between debate and reflection.

In this project, we have embarked on a journey to explore the potential of a debating interaction framework among LLMs. With MAD, the nature of agents being in the state of 'tit for tat' determines that (1) the distorted thinking of one agent can be corrected by the other one 😀; (2) the resistance to change of one agent will be complemented by the other one 😄; and (3) either agent can provide external feedback for each other 😆.

Obviously, MAD is less likely to have the DoT issue and can exploit more potential of LLMs. Experiments show that MAD brings significant and consistent improvements on Counterintuitive QA and Commonsense-MT tasks.

JOIN US on this journey of exploring the interaction and debating capability with LLMs. 🚀🚀🚀

Figure 2: Framework of Multi-Agent Debate. Here we designate the devil () as the affirmative side while the angel () as the negative side. We want the angel to correct the devil’s mistakes..

Preparation

pip3 install -r requirements.txt

• Set your openai API_KEY in debate4tran.sh
• Set your openai API_KEY in interactive.py

Run MAD

sh debate4tran.sh 

Run Interactive

If you just want to have a try, you can try the interactive script on your PC.

python3 interactive.py

Or simply try our demo for translation here.

Table 1: Reasoning accuracy on Counter-Intuitive AR.

When Alice walks up the hill, her speed is 1 m/s and when she goes down the hill, her speed is 3 m/s. Then when Alice walks up and down the hill, what is her average speed? (1.5m/s)

Figure 3: An Animation to Show the Process of MAD.

Table 2: Translation performance on Common MT.

Given the Chinese sentence "吃掉敌人一个师。", please provide its translation in English.

• : I believe the correct translation of the Chinese text "吃掉敌人一个师" is "Eat up an enemy division.” The word "吃掉" literally means "eat up" or “consume."
• : I disagree with your translation. I believe the correct translation of the Chinese text "吃掉敌人一个师" is "Eliminate an enemy division.” While "吃掉" can be translated as "eat up" or "consume," it can also mean "eliminate" or "annihilate" in a military context. In this case, it is more appropriate to use "eliminate" as it conveys the idea of completely destroying the enemy division.
• : Both sides presented valid arguments and translations that accurately convey the meaning of the original Chinese text. The choice between 'eat up' and 'eliminate' may depend on the context and tone of the text. Therefore, there is no clear preference for either side at this time.
• : While "eliminate" does convey the idea of completely destroying the enemy division, it doesn't capture the connotation of consuming or devouring the enemy that is implied by “吃掉."
• : In a military context, "吃掉" is often used to describe the complete destruction of the enemy, rather than the consumption of their resources or territory.
• : Both sides presented valid arguments, but the negative side's argument that 'eliminate' better captures the intensity and seriousness of the military language used in the original Chinese text was more convincing. Additionally, 'eliminate' accurately conveys the idea of completely destroying the enemy division. Therefore, the correct English translation of the Chinese text '吃掉敌人一个师' is 'Eliminate an enemy division.'

• 0-Shot CoT: Large Language Models are Zero-Shot Reasoners (NeurIPS 2022)
• Self-Consist: Self-Consistency Improves Chain of Thought Reasoning in Language Models (ICLR 2023)
• Self-Reflect: Reflexion: an autonomous agent with dynamic memory and self-reflection (arxiv 2023)
• MAPS: Exploring Human-Like Translation Strategy with Large Language Models (arxiv 2023)

@article{liang2023encouraging,
  title={Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate},
  author={Liang, Tian and He, Zhiwei and Jiao, Wenxiang and Wang, Xing and Wang, Yan and Wang, Rui and Yang, Yujiu and Tu, Zhaopeng and Shi, Shuming},
  journal={arXiv preprint arXiv:2305.19118},
  year={2023}
}