[TUTORIAL] Create a synthetic dataset with Mistral and distilabel

[sdiazlor]

In this tutorial, we will generate some instructions using the self-instruct approach and then will generate two possible answers using MistralAI models. Then, we will use a higher-level model to judge the answers (mistral-large). Finally, we will use the argilla package to analyze the dataset and push it to HF.

• Runnable in Colab
• Added to README

Thanks in advance @sophiamyang

[gabrielmbmb]

LGTM! Thanks for pulling this @sdiazlor

[sophiamyang]

Hi @sdiazlor, thanks for the PR! Could you add your notebook to the third-party folder https://github.com/mistralai/cookbook/tree/main/third_party?

[pandora-s-git]

Hi there ! Honnest question, what exactly is the advantage here of using 2 smaller models and judge the answers with Mistral Large? Is it really cheaper or/and better than just generating with mistral large directly a dataset without a Judge? I do understand that for DPO you are required to have two possible answers and get the best one, but will this dataset be better than one generated directly by Large?

[sdiazlor]

@pandora-s-git Thanks for your question. The main issue is that DPO highly helps the model to generate answers that are aligned with people's preferences. A basic QA dataset does not provide insights into the model of what's better or worse, and diversity is reduced. That said, using two small models usually is cheaper, sometimes free if they are open source, and their answers usually have a good quality, the larger model here will help not only to create the DPO dataset but also for the annotators (which is highly recommended to obtain a better high-quality dataset) to make their decisions. However, other alignment approaches have also appeared and they require less data as KTO or DOVE. If you want, this blog may be interesting for you.

[sdiazlor]

@sophiamyang Notebook moved to the third_party folder.

[pandora-s-git]

@pandora-s-git Thanks for your question. The main issue is that DPO highly helps the model to generate answers that are aligned with people's preferences. A basic QA dataset does not provide insights into the model of what's better or worse, and diversity is reduced. That said, using two small models usually is cheaper, sometimes free if they are open source, and their answers usually have a good quality, the larger model here will help not only to create the DPO dataset but also for the annotators (which is highly recommended to obtain a better high-quality dataset) to make their decisions. However, other alignment approaches have also appeared and they require less data as KTO or DOVE. If you want, this blog may be interesting for you.

Yeah after some thinking on my own I figured it actually made sense seeing how DPO usually works, thanks for the blog tho will check it out!!