mistral_finetune_api.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "urSaPCZoqEmM"
      },
      "source": [
        "# Mistral Fine-tuning API\n",
        "\n",
        "Check out the docs: https://docs.mistral.ai/capabilities/finetuning/"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "!pip install mistralai pandas"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "27PpM60GqRvR"
      },
      "source": [
        "## Prepare the dataset\n",
        "\n",
        "In this example, let’s use the ultrachat_200k dataset. We load a chunk of the data into Pandas Dataframes, split the data into training and validation, and save the data into the required jsonl format for fine-tuning."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "MUBRITvX59kC"
      },
      "outputs": [],
      "source": [
        "import pandas as pd\n",
        "df = pd.read_parquet('https://huggingface.co/datasets/HuggingFaceH4/ultrachat_200k/resolve/main/data/test_gen-00000-of-00001-3d4cd8309148a71f.parquet')\n",
        "\n",
        "df_train=df.sample(frac=0.995,random_state=200)\n",
        "df_eval=df.drop(df_train.index)\n",
        "\n",
        "df_train.to_json(\"ultrachat_chunk_train.jsonl\", orient=\"records\", lines=True)\n",
        "df_eval.to_json(\"ultrachat_chunk_eval.jsonl\", orient=\"records\", lines=True)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "S1Pv9zx96CR5",
        "outputId": "e412c365-5887-423a-83b0-1a65b6294ba2"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "total 147M\n",
            "drwxr-xr-x 1 root root 4.0K May 31 13:30 sample_data\n",
            "-rw-r--r-- 1 root root 698K Jun  4 09:06 ultrachat_chunk_eval.jsonl\n",
            "-rw-r--r-- 1 root root 146M Jun  4 09:06 ultrachat_chunk_train.jsonl\n"
          ]
        }
      ],
      "source": [
        "!ls -lh"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dVfIMwgUJF91"
      },
      "source": [
        "## Reformat dataset\n",
        "If you upload this ultrachat_chunk_train.jsonl to Mistral API, you might encounter an error message “Invalid file format” due to data formatting issues. To reformat the data into the correct format, you can download the reformat_dataset.py script and use it to validate and reformat both the training and evaluation data:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "kaLd2n726tAI",
        "outputId": "edb440bc-bc60-40ae-e284-4869812b037e"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "--2024-06-04 09:07:07--  https://raw.githubusercontent.com/mistralai/mistral-finetune/main/utils/reformat_data.py\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.110.133, 185.199.109.133, 185.199.108.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.110.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 3378 (3.3K) [text/plain]\n",
            "Saving to: ‘reformat_data.py’\n",
            "\n",
            "\rreformat_data.py      0%[                    ]       0  --.-KB/s               \rreformat_data.py    100%[===================>]   3.30K  --.-KB/s    in 0s      \n",
            "\n",
            "2024-06-04 09:07:07 (37.1 MB/s) - ‘reformat_data.py’ saved [3378/3378]\n",
            "\n"
          ]
        }
      ],
      "source": [
        "# download the validation and reformat script\n",
        "!wget https://raw.githubusercontent.com/mistralai/mistral-finetune/main/utils/reformat_data.py"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "QvLwYwfl6vfc",
        "outputId": "bef86874-67fa-4758-a633-1a6e17854b63"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Skip 3674th sample\n",
            "Skip 9176th sample\n",
            "Skip 10559th sample\n",
            "Skip 13293th sample\n",
            "Skip 13973th sample\n",
            "Skip 15219th sample\n"
          ]
        }
      ],
      "source": [
        "# validate and reformat the training data\n",
        "!python reformat_data.py ultrachat_chunk_train.jsonl"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "gOKqvJMM6kgU"
      },
      "outputs": [],
      "source": [
        "# validate the reformat the eval data\n",
        "!python reformat_data.py ultrachat_chunk_eval.jsonl"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "XO3RVA197Ont",
        "outputId": "19063479-6745-48b8-b694-79896ac2486d"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "array([{'content': 'What are the dimensions of the cavity, product, and shipping box of the Sharp SMC1662DS microwave?: With innovative features like preset controls, Sensor Cooking and the Carousel® turntable system, the Sharp® SMC1662DS 1.6 cu. Ft. Stainless Steel Carousel Countertop Microwave makes reheating your favorite foods, snacks and beverages easier than ever. Use popcorn and beverage settings for one-touch cooking. Express Cook allows one-touch cooking up to six minutes. The convenient and flexible \"+30 Sec\" key works as both instant start option and allows you to add more time during cooking.\\nThe Sharp SMC1662DS microwave is a bold design statement in any kitchen. The elegant, grey interior and bright white, LED interior lighting complements the stainless steel finish of this premium appliance.\\nCavity Dimensions (w x h x d): 15.5\" x 10.2\" x 17.1\"\\nProduct Dimensions (w x h x d): 21.8\" x 12.8\" x 17.7\"\\nShipping Dimensions (w x h x d) : 24.4\" x 15.0\" x 20.5\"', 'role': 'user'},\n",
              "       {'content': '', 'role': 'assistant'},\n",
              "       {'content': 'Can you help me find out the weight of the Sharp SMC1662DS microwave?', 'role': 'user'},\n",
              "       {'content': 'Yes, the weight of the Sharp SMC1662DS microwave is 36.4 pounds.', 'role': 'assistant'},\n",
              "       {'content': 'Can you tell me if the Sharp SMC1662DS microwave comes with a warranty?', 'role': 'user'},\n",
              "       {'content': 'Yes, the Sharp SMC1662DS microwave comes with a warranty. It has a 1-year limited warranty on parts and labor, a 4-year limited warranty on the magnetron tube, and an 8-year limited warranty on the magnetron tube for certain models. You should check the warranty details provided in the manual for complete information.', 'role': 'assistant'},\n",
              "       {'content': 'Can you remind me of the innovative features of the Sharp SMC1662DS microwave?', 'role': 'user'}],\n",
              "      dtype=object)"
            ]
          },
          "execution_count": 15,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "df_train.iloc[3674]['messages']"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hu_oLukAJect"
      },
      "source": [
        "## Upload dataset"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "B0UlO1Qa7xi3"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "from mistralai.client import MistralClient\n",
        "\n",
        "api_key = os.environ.get(\"MISTRAL_API_KEY\")\n",
        "client = MistralClient(api_key=api_key)\n",
        "\n",
        "with open(\"ultrachat_chunk_train.jsonl\", \"rb\") as f:\n",
        "    ultrachat_chunk_train = client.files.create(file=(\"ultrachat_chunk_train.jsonl\", f))\n",
        "with open(\"ultrachat_chunk_eval.jsonl\", \"rb\") as f:\n",
        "    ultrachat_chunk_eval = client.files.create(file=(\"ultrachat_chunk_eval.jsonl\", f))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ChnYoKhoapES"
      },
      "outputs": [],
      "source": [
        "import json\n",
        "def pprint(obj):\n",
        "    print(json.dumps(obj.dict(), indent=4))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rIAJdJIc9g2q",
        "outputId": "f5fc042e-8c06-473b-a616-536a0c6dd30c"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "{\n",
            "    \"id\": \"872aa786-538f-4510-a7b0-2cffee00b71d\",\n",
            "    \"object\": \"file\",\n",
            "    \"bytes\": 147449694,\n",
            "    \"created_at\": 1717492198,\n",
            "    \"filename\": \"ultrachat_chunk_train.jsonl\",\n",
            "    \"purpose\": \"fine-tune\"\n",
            "}\n"
          ]
        }
      ],
      "source": [
        "pprint(ultrachat_chunk_train)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5uA-Xnp1RTmx",
        "outputId": "ac0bdaba-4af1-4f19-a8ab-dd19482b1943"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "{\n",
            "    \"id\": \"d0ffa827-aca2-42ab-99a8-d449a2cd4da3\",\n",
            "    \"object\": \"file\",\n",
            "    \"bytes\": 691885,\n",
            "    \"created_at\": 1717492199,\n",
            "    \"filename\": \"ultrachat_chunk_eval.jsonl\",\n",
            "    \"purpose\": \"fine-tune\"\n",
            "}\n"
          ]
        }
      ],
      "source": [
        "pprint(ultrachat_chunk_eval)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DqchXMeXJi6U"
      },
      "source": [
        "## Create a fine-tuning job"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "x9Qk42ADRVKo"
      },
      "outputs": [],
      "source": [
        "from mistralai.models.jobs import TrainingParameters\n",
        "\n",
        "created_jobs = client.jobs.create(\n",
        "    model=\"open-mistral-7b\",\n",
        "    training_files=[ultrachat_chunk_train.id],\n",
        "    validation_files=[ultrachat_chunk_eval.id],\n",
        "    hyperparameters=TrainingParameters(\n",
        "        training_steps=10,\n",
        "        learning_rate=0.0001,\n",
        "        )\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "bkZlYvwGanL1",
        "outputId": "396cd040-643b-4296-b026-bb3589df44de"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "{\n",
            "    \"id\": \"63be4026-cb98-4dc0-becd-83c424a42e0e\",\n",
            "    \"hyperparameters\": {\n",
            "        \"training_steps\": 10,\n",
            "        \"learning_rate\": 0.0001\n",
            "    },\n",
            "    \"fine_tuned_model\": null,\n",
            "    \"model\": \"open-mistral-7b\",\n",
            "    \"status\": \"QUEUED\",\n",
            "    \"job_type\": \"FT\",\n",
            "    \"created_at\": 1717492375,\n",
            "    \"modified_at\": 1717492376,\n",
            "    \"training_files\": [\n",
            "        \"872aa786-538f-4510-a7b0-2cffee00b71d\"\n",
            "    ],\n",
            "    \"validation_files\": [\n",
            "        \"d0ffa827-aca2-42ab-99a8-d449a2cd4da3\"\n",
            "    ],\n",
            "    \"object\": \"job\",\n",
            "    \"integrations\": []\n",
            "}\n"
          ]
        }
      ],
      "source": [
        "pprint(created_jobs)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "OQXIE2G3c-Ds",
        "outputId": "83b955b3-5666-4b23-abbe-fa4dbe0ec676"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "{\n",
            "    \"id\": \"63be4026-cb98-4dc0-becd-83c424a42e0e\",\n",
            "    \"hyperparameters\": {\n",
            "        \"training_steps\": 10,\n",
            "        \"learning_rate\": 0.0001\n",
            "    },\n",
            "    \"fine_tuned_model\": null,\n",
            "    \"model\": \"open-mistral-7b\",\n",
            "    \"status\": \"RUNNING\",\n",
            "    \"job_type\": \"FT\",\n",
            "    \"created_at\": 1717492375,\n",
            "    \"modified_at\": 1717492377,\n",
            "    \"training_files\": [\n",
            "        \"872aa786-538f-4510-a7b0-2cffee00b71d\"\n",
            "    ],\n",
            "    \"validation_files\": [\n",
            "        \"d0ffa827-aca2-42ab-99a8-d449a2cd4da3\"\n",
            "    ],\n",
            "    \"object\": \"job\",\n",
            "    \"integrations\": [],\n",
            "    \"events\": [\n",
            "        {\n",
            "            \"name\": \"status-updated\",\n",
            "            \"data\": {\n",
            "                \"status\": \"RUNNING\"\n",
            "            },\n",
            "            \"created_at\": 1717492377\n",
            "        },\n",
            "        {\n",
            "            \"name\": \"status-updated\",\n",
            "            \"data\": {\n",
            "                \"status\": \"QUEUED\"\n",
            "            },\n",
            "            \"created_at\": 1717492375\n",
            "        }\n",
            "    ],\n",
            "    \"checkpoints\": []\n",
            "}\n",
            "Job is RUNNING, waiting 10 seconds\n",
            "{\n",
            "    \"id\": \"63be4026-cb98-4dc0-becd-83c424a42e0e\",\n",
            "    \"hyperparameters\": {\n",
            "        \"training_steps\": 10,\n",
            "        \"learning_rate\": 0.0001\n",
            "    },\n",
            "    \"fine_tuned_model\": null,\n",
            "    \"model\": \"open-mistral-7b\",\n",
            "    \"status\": \"RUNNING\",\n",
            "    \"job_type\": \"FT\",\n",
            "    \"created_at\": 1717492375,\n",
            "    \"modified_at\": 1717492377,\n",
            "    \"training_files\": [\n",
            "        \"872aa786-538f-4510-a7b0-2cffee00b71d\"\n",
            "    ],\n",
            "    \"validation_files\": [\n",
            "        \"d0ffa827-aca2-42ab-99a8-d449a2cd4da3\"\n",
            "    ],\n",
            "    \"object\": \"job\",\n",
            "    \"integrations\": [],\n",
            "    \"events\": [\n",
            "        {\n",
            "            \"name\": \"status-updated\",\n",
            "            \"data\": {\n",
            "                \"status\": \"RUNNING\"\n",
            "            },\n",
            "            \"created_at\": 1717492377\n",
            "        },\n",
            "        {\n",
            "            \"name\": \"status-updated\",\n",
            "            \"data\": {\n",
            "                \"status\": \"QUEUED\"\n",
            "            },\n",
            "            \"created_at\": 1717492375\n",
            "        }\n",
            "    ],\n",
            "    \"checkpoints\": []\n",
            "}\n",
            "Job is RUNNING, waiting 10 seconds\n",
            "{\n",
            "    \"id\": \"63be4026-cb98-4dc0-becd-83c424a42e0e\",\n",
            "    \"hyperparameters\": {\n",
            "        \"training_steps\": 10,\n",
            "        \"learning_rate\": 0.0001\n",
            "    },\n",
            "    \"fine_tuned_model\": null,\n",
            "    \"model\": \"open-mistral-7b\",\n",
            "    \"status\": \"RUNNING\",\n",
            "    \"job_type\": \"FT\",\n",
            "    \"created_at\": 1717492375,\n",
            "    \"modified_at\": 1717492377,\n",
            "    \"training_files\": [\n",
            "        \"872aa786-538f-4510-a7b0-2cffee00b71d\"\n",
            "    ],\n",
            "    \"validation_files\": [\n",
            "        \"d0ffa827-aca2-42ab-99a8-d449a2cd4da3\"\n",
            "    ],\n",
            "    \"object\": \"job\",\n",
            "    \"integrations\": [],\n",
            "    \"events\": [\n",
            "        {\n",
            "            \"name\": \"status-updated\",\n",
            "            \"data\": {\n",
            "                \"status\": \"RUNNING\"\n",
            "            },\n",
            "            \"created_at\": 1717492377\n",
            "        },\n",
            "        {\n",
            "            \"name\": \"status-updated\",\n",
            "            \"data\": {\n",
            "                \"status\": \"QUEUED\"\n",
            "            },\n",
            "            \"created_at\": 1717492375\n",
            "        }\n",
            "    ],\n",
            "    \"checkpoints\": []\n",
            "}\n",
            "Job is RUNNING, waiting 10 seconds\n",
            "{\n",
            "    \"id\": \"63be4026-cb98-4dc0-becd-83c424a42e0e\",\n",
            "    \"hyperparameters\": {\n",
            "        \"training_steps\": 10,\n",
            "        \"learning_rate\": 0.0001\n",
            "    },\n",
            "    \"fine_tuned_model\": null,\n",
            "    \"model\": \"open-mistral-7b\",\n",
            "    \"status\": \"RUNNING\",\n",
            "    \"job_type\": \"FT\",\n",
            "    \"created_at\": 1717492375,\n",
            "    \"modified_at\": 1717492377,\n",
            "    \"training_files\": [\n",
            "        \"872aa786-538f-4510-a7b0-2cffee00b71d\"\n",
            "    ],\n",
            "    \"validation_files\": [\n",
            "        \"d0ffa827-aca2-42ab-99a8-d449a2cd4da3\"\n",
            "    ],\n",
            "    \"object\": \"job\",\n",
            "    \"integrations\": [],\n",
            "    \"events\": [\n",
            "        {\n",
            "            \"name\": \"status-updated\",\n",
            "            \"data\": {\n",
            "                \"status\": \"RUNNING\"\n",
            "            },\n",
            "            \"created_at\": 1717492377\n",
            "        },\n",
            "        {\n",
            "            \"name\": \"status-updated\",\n",
            "            \"data\": {\n",
            "                \"status\": \"QUEUED\"\n",
            "            },\n",
            "            \"created_at\": 1717492375\n",
            "        }\n",
            "    ],\n",
            "    \"checkpoints\": []\n",
            "}\n",
            "Job is RUNNING, waiting 10 seconds\n",
            "{\n",
            "    \"id\": \"63be4026-cb98-4dc0-becd-83c424a42e0e\",\n",
            "    \"hyperparameters\": {\n",
            "        \"training_steps\": 10,\n",
            "        \"learning_rate\": 0.0001\n",
            "    },\n",
            "    \"fine_tuned_model\": null,\n",
            "    \"model\": \"open-mistral-7b\",\n",
            "    \"status\": \"RUNNING\",\n",
            "    \"job_type\": \"FT\",\n",
            "    \"created_at\": 1717492375,\n",
            "    \"modified_at\": 1717492377,\n",
            "    \"training_files\": [\n",
            "        \"872aa786-538f-4510-a7b0-2cffee00b71d\"\n",
            "    ],\n",
            "    \"validation_files\": [\n",
            "        \"d0ffa827-aca2-42ab-99a8-d449a2cd4da3\"\n",
            "    ],\n",
            "    \"object\": \"job\",\n",
            "    \"integrations\": [],\n",
            "    \"events\": [\n",
            "        {\n",
            "            \"name\": \"status-updated\",\n",
            "            \"data\": {\n",
            "                \"status\": \"RUNNING\"\n",
            "            },\n",
            "            \"created_at\": 1717492377\n",
            "        },\n",
            "        {\n",
            "            \"name\": \"status-updated\",\n",
            "            \"data\": {\n",
            "                \"status\": \"QUEUED\"\n",
            "            },\n",
            "            \"created_at\": 1717492375\n",
            "        }\n",
            "    ],\n",
            "    \"checkpoints\": []\n",
            "}\n",
            "Job is RUNNING, waiting 10 seconds\n",
            "{\n",
            "    \"id\": \"63be4026-cb98-4dc0-becd-83c424a42e0e\",\n",
            "    \"hyperparameters\": {\n",
            "        \"training_steps\": 10,\n",
            "        \"learning_rate\": 0.0001\n",
            "    },\n",
            "    \"fine_tuned_model\": null,\n",
            "    \"model\": \"open-mistral-7b\",\n",
            "    \"status\": \"RUNNING\",\n",
            "    \"job_type\": \"FT\",\n",
            "    \"created_at\": 1717492375,\n",
            "    \"modified_at\": 1717492377,\n",
            "    \"training_files\": [\n",
            "        \"872aa786-538f-4510-a7b0-2cffee00b71d\"\n",
            "    ],\n",
            "    \"validation_files\": [\n",
            "        \"d0ffa827-aca2-42ab-99a8-d449a2cd4da3\"\n",
            "    ],\n",
            "    \"object\": \"job\",\n",
            "    \"integrations\": [],\n",
            "    \"events\": [\n",
            "        {\n",
            "            \"name\": \"status-updated\",\n",
            "            \"data\": {\n",
            "                \"status\": \"RUNNING\"\n",
            "            },\n",
            "            \"created_at\": 1717492377\n",
            "        },\n",
            "        {\n",
            "            \"name\": \"status-updated\",\n",
            "            \"data\": {\n",
            "                \"status\": \"QUEUED\"\n",
            "            },\n",
            "            \"created_at\": 1717492375\n",
            "        }\n",
            "    ],\n",
            "    \"checkpoints\": []\n",
            "}\n",
            "Job is RUNNING, waiting 10 seconds\n",
            "{\n",
            "    \"id\": \"63be4026-cb98-4dc0-becd-83c424a42e0e\",\n",
            "    \"hyperparameters\": {\n",
            "        \"training_steps\": 10,\n",
            "        \"learning_rate\": 0.0001\n",
            "    },\n",
            "    \"fine_tuned_model\": null,\n",
            "    \"model\": \"open-mistral-7b\",\n",
            "    \"status\": \"RUNNING\",\n",
            "    \"job_type\": \"FT\",\n",
            "    \"created_at\": 1717492375,\n",
            "    \"modified_at\": 1717492377,\n",
            "    \"training_files\": [\n",
            "        \"872aa786-538f-4510-a7b0-2cffee00b71d\"\n",
            "    ],\n",
            "    \"validation_files\": [\n",
            "        \"d0ffa827-aca2-42ab-99a8-d449a2cd4da3\"\n",
            "    ],\n",
            "    \"object\": \"job\",\n",
            "    \"integrations\": [],\n",
            "    \"events\": [\n",
            "        {\n",
            "            \"name\": \"status-updated\",\n",
            "            \"data\": {\n",
            "                \"status\": \"RUNNING\"\n",
            "            },\n",
            "            \"created_at\": 1717492377\n",
            "        },\n",
            "        {\n",
            "            \"name\": \"status-updated\",\n",
            "            \"data\": {\n",
            "                \"status\": \"QUEUED\"\n",
            "            },\n",
            "            \"created_at\": 1717492375\n",
            "        }\n",
            "    ],\n",
            "    \"checkpoints\": [\n",
            "        {\n",
            "            \"metrics\": {\n",
            "                \"train_loss\": 0.890919,\n",
            "                \"valid_loss\": 0.866454,\n",
            "                \"valid_mean_token_accuracy\": 1.823177\n",
            "            },\n",
            "            \"step_number\": 10,\n",
            "            \"created_at\": 1717492439\n",
            "        }\n",
            "    ]\n",
            "}\n",
            "Job is RUNNING, waiting 10 seconds\n",
            "{\n",
            "    \"id\": \"63be4026-cb98-4dc0-becd-83c424a42e0e\",\n",
            "    \"hyperparameters\": {\n",
            "        \"training_steps\": 10,\n",
            "        \"learning_rate\": 0.0001\n",
            "    },\n",
            "    \"fine_tuned_model\": \"ft:open-mistral-7b:8e2706f0:20240604:63be4026\",\n",
            "    \"model\": \"open-mistral-7b\",\n",
            "    \"status\": \"SUCCESS\",\n",
            "    \"job_type\": \"FT\",\n",
            "    \"created_at\": 1717492375,\n",
            "    \"modified_at\": 1717492452,\n",
            "    \"training_files\": [\n",
            "        \"872aa786-538f-4510-a7b0-2cffee00b71d\"\n",
            "    ],\n",
            "    \"validation_files\": [\n",
            "        \"d0ffa827-aca2-42ab-99a8-d449a2cd4da3\"\n",
            "    ],\n",
            "    \"object\": \"job\",\n",
            "    \"integrations\": [],\n",
            "    \"events\": [\n",
            "        {\n",
            "            \"name\": \"status-updated\",\n",
            "            \"data\": {\n",
            "                \"status\": \"SUCCESS\"\n",
            "            },\n",
            "            \"created_at\": 1717492452\n",
            "        },\n",
            "        {\n",
            "            \"name\": \"status-updated\",\n",
            "            \"data\": {\n",
            "                \"status\": \"RUNNING\"\n",
            "            },\n",
            "            \"created_at\": 1717492377\n",
            "        },\n",
            "        {\n",
            "            \"name\": \"status-updated\",\n",
            "            \"data\": {\n",
            "                \"status\": \"QUEUED\"\n",
            "            },\n",
            "            \"created_at\": 1717492375\n",
            "        }\n",
            "    ],\n",
            "    \"checkpoints\": [\n",
            "        {\n",
            "            \"metrics\": {\n",
            "                \"train_loss\": 0.890919,\n",
            "                \"valid_loss\": 0.866454,\n",
            "                \"valid_mean_token_accuracy\": 1.823177\n",
            "            },\n",
            "            \"step_number\": 10,\n",
            "            \"created_at\": 1717492439\n",
            "        }\n",
            "    ]\n",
            "}\n",
            "Job is SUCCESS, waiting 10 seconds\n"
          ]
        }
      ],
      "source": [
        "import time\n",
        "\n",
        "retrieved_job = client.jobs.retrieve(created_jobs.id)\n",
        "while retrieved_job.status in [\"RUNNING\", \"QUEUED\"]:\n",
        "    retrieved_job = client.jobs.retrieve(created_jobs.id)\n",
        "    pprint(retrieved_job)\n",
        "    print(f\"Job is {retrieved_job.status}, waiting 10 seconds\")\n",
        "    time.sleep(10)\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_x6wRaDtXDzt",
        "outputId": "1efa4948-0ca8-4bc1-85d2-7b72b7e4a931"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "{\n",
            "    \"data\": [\n",
            "        {\n",
            "            \"id\": \"63be4026-cb98-4dc0-becd-83c424a42e0e\",\n",
            "            \"hyperparameters\": {\n",
            "                \"training_steps\": 10,\n",
            "                \"learning_rate\": 0.0001\n",
            "            },\n",
            "            \"fine_tuned_model\": \"ft:open-mistral-7b:8e2706f0:20240604:63be4026\",\n",
            "            \"model\": \"open-mistral-7b\",\n",
            "            \"status\": \"SUCCESS\",\n",
            "            \"job_type\": \"FT\",\n",
            "            \"created_at\": 1717492375,\n",
            "            \"modified_at\": 1717492452,\n",
            "            \"training_files\": [\n",
            "                \"872aa786-538f-4510-a7b0-2cffee00b71d\"\n",
            "            ],\n",
            "            \"validation_files\": [\n",
            "                \"d0ffa827-aca2-42ab-99a8-d449a2cd4da3\"\n",
            "            ],\n",
            "            \"object\": \"job\",\n",
            "            \"integrations\": []\n",
            "        },\n",
            "        {\n",
            "            \"id\": \"54e86210-7eee-4486-a47f-a00c84880b02\",\n",
            "            \"hyperparameters\": {\n",
            "                \"training_steps\": 100,\n",
            "                \"learning_rate\": 0.0001\n",
            "            },\n",
            "            \"fine_tuned_model\": null,\n",
            "            \"model\": \"open-mistral-7b\",\n",
            "            \"status\": \"RUNNING\",\n",
            "            \"job_type\": \"FT\",\n",
            "            \"created_at\": 1717492217,\n",
            "            \"modified_at\": 1717492217,\n",
            "            \"training_files\": [\n",
            "                \"872aa786-538f-4510-a7b0-2cffee00b71d\"\n",
            "            ],\n",
            "            \"validation_files\": [\n",
            "                \"d0ffa827-aca2-42ab-99a8-d449a2cd4da3\"\n",
            "            ],\n",
            "            \"object\": \"job\",\n",
            "            \"integrations\": []\n",
            "        },\n",
            "        {\n",
            "            \"id\": \"3dec82fa-3466-48a5-8f01-729489217bcc\",\n",
            "            \"hyperparameters\": {\n",
            "                \"training_steps\": 80,\n",
            "                \"learning_rate\": 0.0001\n",
            "            },\n",
            "            \"fine_tuned_model\": null,\n",
            "            \"model\": \"mistral-small-latest\",\n",
            "            \"status\": \"FAILED\",\n",
            "            \"job_type\": \"FT\",\n",
            "            \"created_at\": 1717434108,\n",
            "            \"modified_at\": 1717434300,\n",
            "            \"training_files\": [\n",
            "                \"33c5a6b7-dcff-4a2c-ad92-44b72cb826a5\"\n",
            "            ],\n",
            "            \"validation_files\": [\n",
            "                \"747d2b0b-5a6e-4bc0-898d-c444c7e52b8f\"\n",
            "            ],\n",
            "            \"object\": \"job\",\n",
            "            \"integrations\": [\n",
            "                {\n",
            "                    \"type\": \"wandb\",\n",
            "                    \"project\": \"small_SlackStral\",\n",
            "                    \"name\": null,\n",
            "                    \"run_name\": \"run_3_split_big\"\n",
            "                }\n",
            "            ]\n",
            "        },\n",
            "        {\n",
            "            \"id\": \"26905786-c0c7-4e93-907f-78ad79c77a41\",\n",
            "            \"hyperparameters\": {\n",
            "                \"training_steps\": 500,\n",
            "                \"learning_rate\": 0.0001\n",
            "            },\n",
            "            \"fine_tuned_model\": \"ft:mistral-small-latest:8e2706f0:20240603:26905786\",\n",
            "            \"model\": \"mistral-small-latest\",\n",
            "            \"status\": \"SUCCESS\",\n",
            "            \"job_type\": \"FT\",\n",
            "            \"created_at\": 1717430336,\n",
            "            \"modified_at\": 1717432905,\n",
            "            \"training_files\": [\n",
            "                \"951ce495-1d1d-46b2-82d4-0b55d80a3cfd\"\n",
            "            ],\n",
            "            \"validation_files\": [],\n",
            "            \"object\": \"job\",\n",
            "            \"integrations\": [\n",
            "                {\n",
            "                    \"type\": \"wandb\",\n",
            "                    \"project\": \"test_api\",\n",
            "                    \"name\": null,\n",
            "                    \"run_name\": \"abstractstral\"\n",
            "                }\n",
            "            ]\n",
            "        },\n",
            "        {\n",
            "            \"id\": \"b611179e-c5de-4c71-adbd-2f6e9d209903\",\n",
            "            \"hyperparameters\": {\n",
            "                \"training_steps\": 100,\n",
            "                \"learning_rate\": 0.0001\n",
            "            },\n",
            "            \"fine_tuned_model\": \"ft:mistral-small-latest:8e2706f0:20240603:b611179e\",\n",
            "            \"model\": \"mistral-small-latest\",\n",
            "            \"status\": \"SUCCESS\",\n",
            "            \"job_type\": \"FT\",\n",
            "            \"created_at\": 1717430161,\n",
            "            \"modified_at\": 1717430803,\n",
            "            \"training_files\": [\n",
            "                \"951ce495-1d1d-46b2-82d4-0b55d80a3cfd\"\n",
            "            ],\n",
            "            \"validation_files\": [],\n",
            "            \"object\": \"job\",\n",
            "            \"integrations\": [\n",
            "                {\n",
            "                    \"type\": \"wandb\",\n",
            "                    \"project\": \"test_api\",\n",
            "                    \"name\": null,\n",
            "                    \"run_name\": \"abstractstral\"\n",
            "                }\n",
            "            ]\n",
            "        },\n",
            "        {\n",
            "            \"id\": \"a0f51b5f-9431-4b30-a934-307ebb1fb71a\",\n",
            "            \"hyperparameters\": {\n",
            "                \"training_steps\": 100,\n",
            "                \"learning_rate\": 0.0001\n",
            "            },\n",
            "            \"fine_tuned_model\": null,\n",
            "            \"model\": \"mistral-small-latest\",\n",
            "            \"status\": \"FAILED\",\n",
            "            \"job_type\": \"FT\",\n",
            "            \"created_at\": 1717428505,\n",
            "            \"modified_at\": 1717430243,\n",
            "            \"training_files\": [\n",
            "                \"5723cc21-8a4e-4796-9479-4c1f6d203bec\"\n",
            "            ],\n",
            "            \"validation_files\": [\n",
            "                \"0ace7ae4-9d51-445e-bc50-54d87e5b4fd4\"\n",
            "            ],\n",
            "            \"object\": \"job\",\n",
            "            \"integrations\": [\n",
            "                {\n",
            "                    \"type\": \"wandb\",\n",
            "                    \"project\": \"test_api\",\n",
            "                    \"name\": null,\n",
            "                    \"run_name\": \"abstractstral\"\n",
            "                }\n",
            "            ]\n",
            "        },\n",
            "        {\n",
            "            \"id\": \"018c9dbc-80c3-44d2-8c40-d45e4a8c9115\",\n",
            "            \"hyperparameters\": {\n",
            "                \"training_steps\": 80,\n",
            "                \"learning_rate\": 0.0001\n",
            "            },\n",
            "            \"fine_tuned_model\": \"ft:mistral-small-latest:8e2706f0:20240603:018c9dbc\",\n",
            "            \"model\": \"mistral-small-latest\",\n",
            "            \"status\": \"SUCCESS\",\n",
            "            \"job_type\": \"FT\",\n",
            "            \"created_at\": 1717427445,\n",
            "            \"modified_at\": 1717428010,\n",
            "            \"training_files\": [\n",
            "                \"76696115-090b-4478-8b1f-0cccbcc9e69a\"\n",
            "            ],\n",
            "            \"validation_files\": [\n",
            "                \"2fa5ead2-45fa-4fb9-9445-951947458f30\"\n",
            "            ],\n",
            "            \"object\": \"job\",\n",
            "            \"integrations\": [\n",
            "                {\n",
            "                    \"type\": \"wandb\",\n",
            "                    \"project\": \"small_SlackStral\",\n",
            "                    \"name\": null,\n",
            "                    \"run_name\": \"run_3_split\"\n",
            "                }\n",
            "            ]\n",
            "        },\n",
            "        {\n",
            "            \"id\": \"480c9e19-fc8d-49cf-8763-bbe882b35077\",\n",
            "            \"hyperparameters\": {\n",
            "                \"training_steps\": 1000,\n",
            "                \"learning_rate\": 0.0001\n",
            "            },\n",
            "            \"fine_tuned_model\": null,\n",
            "            \"model\": \"mistral-small-latest\",\n",
            "            \"status\": \"FAILED\",\n",
            "            \"job_type\": \"FT\",\n",
            "            \"created_at\": 1717427336,\n",
            "            \"modified_at\": 1717430228,\n",
            "            \"training_files\": [\n",
            "                \"5723cc21-8a4e-4796-9479-4c1f6d203bec\"\n",
            "            ],\n",
            "            \"validation_files\": [\n",
            "                \"0ace7ae4-9d51-445e-bc50-54d87e5b4fd4\"\n",
            "            ],\n",
            "            \"object\": \"job\",\n",
            "            \"integrations\": [\n",
            "                {\n",
            "                    \"type\": \"wandb\",\n",
            "                    \"project\": \"test_api\",\n",
            "                    \"name\": null,\n",
            "                    \"run_name\": \"abstractstral\"\n",
            "                }\n",
            "            ]\n",
            "        },\n",
            "        {\n",
            "            \"id\": \"f3cc2bf9-0cbb-4120-b35d-ef1d20a96da9\",\n",
            "            \"hyperparameters\": {\n",
            "                \"training_steps\": 40,\n",
            "                \"learning_rate\": 0.0005\n",
            "            },\n",
            "            \"fine_tuned_model\": \"ft:mistral-small-latest:8e2706f0:20240531:f3cc2bf9\",\n",
            "            \"model\": \"mistral-small-latest\",\n",
            "            \"status\": \"SUCCESS\",\n",
            "            \"job_type\": \"FT\",\n",
            "            \"created_at\": 1717174575,\n",
            "            \"modified_at\": 1717174844,\n",
            "            \"training_files\": [\n",
            "                \"8e1b5a29-7f43-44f1-9984-fb8061f73b13\"\n",
            "            ],\n",
            "            \"validation_files\": [\n",
            "                \"58630896-a238-4f80-a20e-0a35c1bc001e\"\n",
            "            ],\n",
            "            \"object\": \"job\",\n",
            "            \"integrations\": [\n",
            "                {\n",
            "                    \"type\": \"wandb\",\n",
            "                    \"project\": \"test_api\",\n",
            "                    \"name\": null,\n",
            "                    \"run_name\": \"menchstral\"\n",
            "                }\n",
            "            ]\n",
            "        },\n",
            "        {\n",
            "            \"id\": \"2303145d-fd09-4969-aeb9-e409cc1b4755\",\n",
            "            \"hyperparameters\": {\n",
            "                \"training_steps\": 20,\n",
            "                \"learning_rate\": 0.001\n",
            "            },\n",
            "            \"fine_tuned_model\": \"ft:mistral-small-latest:8e2706f0:20240531:2303145d\",\n",
            "            \"model\": \"mistral-small-latest\",\n",
            "            \"status\": \"SUCCESS\",\n",
            "            \"job_type\": \"FT\",\n",
            "            \"created_at\": 1717174233,\n",
            "            \"modified_at\": 1717174403,\n",
            "            \"training_files\": [\n",
            "                \"8e1b5a29-7f43-44f1-9984-fb8061f73b13\"\n",
            "            ],\n",
            "            \"validation_files\": [\n",
            "                \"58630896-a238-4f80-a20e-0a35c1bc001e\"\n",
            "            ],\n",
            "            \"object\": \"job\",\n",
            "            \"integrations\": [\n",
            "                {\n",
            "                    \"type\": \"wandb\",\n",
            "                    \"project\": \"test_api\",\n",
            "                    \"name\": null,\n",
            "                    \"run_name\": \"menchstral\"\n",
            "                }\n",
            "            ]\n",
            "        },\n",
            "        {\n",
            "            \"id\": \"df895bef-7cea-4091-98bf-d65ea387d50a\",\n",
            "            \"hyperparameters\": {\n",
            "                \"training_steps\": 10,\n",
            "                \"learning_rate\": 0.0001\n",
            "            },\n",
            "            \"fine_tuned_model\": \"ft:mistral-small-latest:8e2706f0:20240531:df895bef\",\n",
            "            \"model\": \"mistral-small-latest\",\n",
            "            \"status\": \"SUCCESS\",\n",
            "            \"job_type\": \"FT\",\n",
            "            \"created_at\": 1717173957,\n",
            "            \"modified_at\": 1717174071,\n",
            "            \"training_files\": [\n",
            "                \"8e1b5a29-7f43-44f1-9984-fb8061f73b13\"\n",
            "            ],\n",
            "            \"validation_files\": [\n",
            "                \"58630896-a238-4f80-a20e-0a35c1bc001e\"\n",
            "            ],\n",
            "            \"object\": \"job\",\n",
            "            \"integrations\": [\n",
            "                {\n",
            "                    \"type\": \"wandb\",\n",
            "                    \"project\": \"test_api\",\n",
            "                    \"name\": null,\n",
            "                    \"run_name\": \"menchstral\"\n",
            "                }\n",
            "            ]\n",
            "        }\n",
            "    ],\n",
            "    \"object\": \"list\"\n",
            "}\n"
          ]
        }
      ],
      "source": [
        "# List jobs\n",
        "jobs = client.jobs.list()\n",
        "pprint(jobs)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "OhTWA5uJXHNp",
        "outputId": "a6db9934-a231-4650-d539-17c25e32b8d5"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "{\n",
            "    \"id\": \"63be4026-cb98-4dc0-becd-83c424a42e0e\",\n",
            "    \"hyperparameters\": {\n",
            "        \"training_steps\": 10,\n",
            "        \"learning_rate\": 0.0001\n",
            "    },\n",
            "    \"fine_tuned_model\": \"ft:open-mistral-7b:8e2706f0:20240604:63be4026\",\n",
            "    \"model\": \"open-mistral-7b\",\n",
            "    \"status\": \"SUCCESS\",\n",
            "    \"job_type\": \"FT\",\n",
            "    \"created_at\": 1717492375,\n",
            "    \"modified_at\": 1717492452,\n",
            "    \"training_files\": [\n",
            "        \"872aa786-538f-4510-a7b0-2cffee00b71d\"\n",
            "    ],\n",
            "    \"validation_files\": [\n",
            "        \"d0ffa827-aca2-42ab-99a8-d449a2cd4da3\"\n",
            "    ],\n",
            "    \"object\": \"job\",\n",
            "    \"integrations\": [],\n",
            "    \"events\": [\n",
            "        {\n",
            "            \"name\": \"status-updated\",\n",
            "            \"data\": {\n",
            "                \"status\": \"SUCCESS\"\n",
            "            },\n",
            "            \"created_at\": 1717492452\n",
            "        },\n",
            "        {\n",
            "            \"name\": \"status-updated\",\n",
            "            \"data\": {\n",
            "                \"status\": \"RUNNING\"\n",
            "            },\n",
            "            \"created_at\": 1717492377\n",
            "        },\n",
            "        {\n",
            "            \"name\": \"status-updated\",\n",
            "            \"data\": {\n",
            "                \"status\": \"QUEUED\"\n",
            "            },\n",
            "            \"created_at\": 1717492375\n",
            "        }\n",
            "    ],\n",
            "    \"checkpoints\": [\n",
            "        {\n",
            "            \"metrics\": {\n",
            "                \"train_loss\": 0.890919,\n",
            "                \"valid_loss\": 0.866454,\n",
            "                \"valid_mean_token_accuracy\": 1.823177\n",
            "            },\n",
            "            \"step_number\": 10,\n",
            "            \"created_at\": 1717492439\n",
            "        }\n",
            "    ]\n",
            "}\n"
          ]
        }
      ],
      "source": [
        "# Retrieve a jobs\n",
        "retrieved_jobs = client.jobs.retrieve(created_jobs.id)\n",
        "pprint(retrieved_jobs)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LK-cSS2EJv-e"
      },
      "source": [
        "## Use a fine-tuned model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "M8vRIB4yXQ5a"
      },
      "outputs": [],
      "source": [
        "from mistralai.models.chat_completion import ChatMessage\n",
        "\n",
        "chat_response = client.chat(\n",
        "    model=retrieved_jobs.fine_tuned_model,\n",
        "    messages=[ChatMessage(role='user', content='What is the best French cheese?')]\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "hp0oj5jvX0dP",
        "outputId": "975ed74b-25fe-4e69-c6f8-0a61a69b5a8b"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "{\n",
            "    \"id\": \"a0b12a4ba1b9402ba8fe541526f26749\",\n",
            "    \"object\": \"chat.completion\",\n",
            "    \"created\": 1717492478,\n",
            "    \"model\": \"ft:open-mistral-7b:8e2706f0:20240604:63be4026\",\n",
            "    \"choices\": [\n",
            "        {\n",
            "            \"index\": 0,\n",
            "            \"message\": {\n",
            "                \"role\": \"assistant\",\n",
            "                \"content\": \"It's impossible to definitively say which is the best French cheese, as it depends on one's personal taste and preferences. Some popular French cheeses include Camembert, Brie, Comt\\u00e9, and Roquefort. These cheeses vary in flavor, texture, and intensity, so it is best to try a few to determine which you prefer.\",\n",
            "                \"name\": null,\n",
            "                \"tool_calls\": null,\n",
            "                \"tool_call_id\": null\n",
            "            },\n",
            "            \"finish_reason\": \"stop\"\n",
            "        }\n",
            "    ],\n",
            "    \"usage\": {\n",
            "        \"prompt_tokens\": 10,\n",
            "        \"total_tokens\": 87,\n",
            "        \"completion_tokens\": 77\n",
            "    }\n",
            "}\n"
          ]
        }
      ],
      "source": [
        "pprint(chat_response)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1xA68sx-J1Jj"
      },
      "source": [
        "## Integration with Weights and Biases\n",
        "We can also offer support for integration with Weights & Biases (W&B) to monitor and track various metrics and statistics associated with our fine-tuning jobs. To enable integration with W&B, you will need to create an account with W&B and add your W&B information in the “integrations” section in the job creation request:\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "L7ZYQzZ7YcYb"
      },
      "outputs": [],
      "source": [
        "from mistralai.models.jobs import WandbIntegrationIn\n",
        "\n",
        "WANDB_API_KEY = \"XXX\"\n",
        "\n",
        "created_jobs = client.jobs.create(\n",
        "    model=\"open-mistral-7b\",\n",
        "    training_files=[ultrachat_chunk_train.id],\n",
        "    validation_files=[ultrachat_chunk_eval.id],\n",
        "    hyperparameters=TrainingParameters(\n",
        "        training_steps=100,\n",
        "        learning_rate=0.0001,\n",
        "    ),\n",
        "    integrations=[\n",
        "        WandbIntegrationIn(\n",
        "            project=\"test_ft_api\",\n",
        "            run_name=\"test\",\n",
        "            api_key=WANDB_API_KEY,\n",
        "        ).dict()\n",
        "    ],\n",
        ")"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}