Add new GPT-4 Turbo (#19)

CHANGELOG.md

@@ -12,6 +12,7 @@
 - REMOVED: Comparison of several Qwen-1.5 models (removed due to poor scores caused by invalid GGUF)
 - Added benchmark evals for Google Gemini 1.0 Pro (latest version as of 17th Feb 2024)
 - Added benchmark evals for Claude 3 models and MistralAI "mistral-large"
+- Added benchmark for the latest OpenAI GPT-4 Turbo ("gpt-4-turbo-2024-04-09")
 
 ### Fixed
 - Changed the wording from "open-source" models to "locally-hosted" models (a more appropriate description)

Project.toml

@@ -23,4 +23,4 @@ Statistics = "10745b16-79ce-11e8-11f9-7d13ad32a3b2"
 TOML = "fa267f1f-6049-4f14-aa54-33bafae1ed76"
 
 [compat]
-PromptingTools = "0.17"
+PromptingTools = "0.19"

README.md

@@ -62,6 +62,7 @@ The latest GPT3.5-Turbo (0125) scores very well - in many cases, it's as good as
 |--------------------------|---------|-------|---------------------|------------------|------------------|------------|
 | claude-3-opus-20240229 | 20.3 | 83.2 | 19.6 | 2 | 329 | 3.9 |
 | claude-3-sonnet-20240229 | 8.7 | 78.8 | 26.2 | 22 | 308 | 0.73 |
+| gpt-4-turbo-2024-04-09 | 10.8 | 75.3 | 29.6 | 38 | 290 | 1.38 |
 | claude-3-haiku-20240307 | 4.0 | 74.9 | 27.2 | 9 | 261 | 0.05 |
 | gpt-4-0125-preview | 30.3 | 74.4 | 30.3 | 39 | 284 | 1.29 |
 | gpt-4-1106-preview | 22.4 | 74.4 | 29.9 | 19 | 142 | 1.21 |
@@ -78,6 +79,7 @@ The latest GPT3.5-Turbo (0125) scores very well - in many cases, it's as good as
 
 
 
+
 Note: From mid-February 2024, "gpt-3.5-turbo" will point to the latest release, "gpt-3.5-turbo-0125" (deprecating the June release).
 
 Same information, but as a bar chart:
@@ -158,13 +160,13 @@ Learnings so far:
 - Never use the "AsIs" prompt (ie, raw task definition). ALWAYS add some context around the language, situation, etc.
 - Even a simple "In Julia, answer XYZ" prompt can be quite effective. Note that the bigger prompts ("CoT" stands for Chain of Thought) might be confusing the smaller models, hence why this prompt is so effective on average.
 
-| Prompt Template  | Elapsed (s, average) | Elapsed (s, median) | Avg. Score (Max 100 pts) | Median Score (Max 100 pts) |
-|-----------------------|----------------------|---------------------|--------------------------|----------------------------|
-|  InJulia | 14.4 | 10.0 | 53.4 | 50.0 |
-|  JuliaExpertAsk | 10.0 | 6.7 | 50.6 | 50.0 |
-|  JuliaRecapTask | 17.5 | 12.6 | 50.1 | 50.0 |
-|  JuliaExpertCoTTask | 15.9 | 11.1 | 46.7 | 50.0 |
-|  JuliaRecapCoTTask | 16.7 | 11.9 | 45.7 | 50.0 |
+| Prompt Template | Elapsed (s, average) | Elapsed (s, median) | Avg. Score (Max 100 pts) | Median Score (Max 100 pts) |
+|--------------------|----------------------|---------------------|--------------------------|----------------------------|
+| InJulia | 14.3 | 10.1 | 53.9 | 50.0 |
+| JuliaExpertAsk |  9.9 | 6.7 | 51.3 | 50.0 |
+| JuliaRecapTask | 17.3 | 12.4 | 50.6 | 50.0 |
+| JuliaExpertCoTTask | 15.7 | 11.0 | 47.4 | 50.0 |
+| JuliaRecapCoTTask | 16.6 | 11.9 | 46.5 | 50.0 |
 
 
 Note: The XML-based templates are tested only for Claude 3 models (Haiku and Sonnet), that's why we remove them from the comparison.

code_generation/data_analysis/add_yearmonth/gpt-4-turbo-2024-04-09/conversation__InJulia__1SHOT__20240410_072013__626.json

@@ -0,0 +1 @@
+[{"content":"Act as a helpful AI assistant","variables":[],"_type":"systemmessage"},{"content":"In Julia, Given a DataFrame `df` with column `dt` representing DateTimes. Write a function `add_yearmonth` that creates a new column `ym` by extracting year and month from `dt` and concatenating them together as an integer in format: “yyyymm”.","variables":[],"_type":"usermessage"},{"content":"To create the function `add_yearmonth` in Julia that adds this new column `ym` to your DataFrame `df`, you need to proceed with the following steps:\n\n1. Ensure you have the `DataFrames` package installed and loaded;\n2. Use functionality from the `Dates` module to work with date and time;\n3. Create a function that extracts year and month from the `DateTime` object and formats them as specified (yyyymm);\n4. Apply this function to create your new column in the DataFrame.\n\nHere’s how you can write this function:\n\n```julia\nusing DataFrames\nimport Dates\n\nfunction add_yearmonth(df)\n # Define a function to convert DateTime to \"yyyymm\"\n function ym_format(dt)\n y = Dates.year(dt) # Extract the year from DateTime\n m = Dates.month(dt) # Extract the month from DateTime\n return 10000 * y + m # Return it as yyyymm integer\n end\n\n # Apply the ym_format function to each element of the 'dt' column\n # and store the results in a new 'ym' column\n df.ym = ym_format.(df.dt)\n\n return df\nend\n```\n\n### Explanation:\n- The `ym_format` function takes a `DateTime` object (`dt`) as input and converts it into an integer representing the year and month in the format `yyyymm`. This is done by multiplying the year by `10000` and adding the month, which effectively places the month right after the year digits.\n- The `.` operator (`ym_format.(df.dt)`) in the expression applies the `ym_format` function element-wise to the entire `dt` column.\n- The new column `ym` is added to the DataFrame `df` with the transformed data.\n\nHere’s an example of how you can use this function:\n\n```julia\nusing DataFrames\n\n# Sample data\ndf = DataFrame(dt=[DateTime(2023, 3, 25, 12, 0), DateTime(2023, 10, 5, 15, 30)])\n\n# Call the function\ndf = add_yearmonth(df)\n\n# View modified DataFrame\nprintln(df)\n```\n\nThe output will display the DataFrame with the newly added `ym` column.","status":200,"tokens":[75,475],"elapsed":10.099480958,"cost":0.015000000000000001,"log_prob":null,"finish_reason":"stop","run_id":1702800130,"sample_id":null,"_type":"aimessage"}]

code_generation/data_analysis/add_yearmonth/gpt-4-turbo-2024-04-09/conversation__InJulia__1SHOT__20240410_072023__463.json

@@ -0,0 +1 @@
+[{"content":"Act as a helpful AI assistant","variables":[],"_type":"systemmessage"},{"content":"In Julia, Given a DataFrame `df` with column `dt` representing DateTimes. Write a function `add_yearmonth` that creates a new column `ym` by extracting year and month from `dt` and concatenating them together as an integer in format: “yyyymm”.","variables":[],"_type":"usermessage"},{"content":"In Julia, you can achieve the requested functionality using the `DataFrames` and `Dates` packages. Here, I'll provide you a step-by-step walk-through on how to implement the function `add_yearmonth` to a DataFrame `df` that contains a `DateTime` column named `dt`. This function will create a new column `ym` where year and month are concatenated into an integer in the format `yyyymm`.\n\nFirst, ensure that you have the `DataFrames` package installed in your Julia environment:\n```julia\nusing Pkg\nPkg.add(\"DataFrames\")\n```\n\nHere’s how you can define the function:\n\n```julia\nusing DataFrames\nusing Dates\n\nfunction add_yearmonth(df::DataFrame)\n # Check if the \"dt\" column exists in the DataFrame\n if :dt in names(df)\n # Create 'ym' by extracting and formatting the year and month from 'dt'\n df.ym = (year.(df.dt) * 100) .+ month.(df.dt)\n return df\n else\n error(\"DataFrame must contain 'dt' column of DateTime type\")\n end\nend\n```\n\nLet's break down some key parts of the function:\n- The `year` and `month` functions from the `Dates` module are used to extract the year and month from each DateTime entry in the `dt` column.\n- We multiply the year by 100 and then add the month to achieve the `yyyymm` format. This computation converts everything to an integer representation.\n- I have used the broadcasting (`.`) to apply the operations across the entire column.\n- There is a check to ensure that the DataFrame contains a column `dt`; if it doesn't, the function will throw an error.\n\n### Example Usage:\nNow, let’s see how to use this function with a DataFrame that includes a `DateTime` column.\n\n```julia\n# Creating a sample DataFrame with DateTime data\ndf = DataFrame(dt = [DateTime(2021, 6, 1), DateTime(2022, 7, 15), DateTime(2023, 1, 23)])\n\n# Applying the function\ndf = add_yearmonth(df)\n\n# Display the modified DataFrame\nprintln(df)\n```\n\nThis will display a DataFrame with an additional column `ym` that shows the year and month combined as requested.\n\nThis approach provides a clear and efficient way to manipulate and extract date-specific information in Julia using DataFrames, and it can easily be adjusted or extended for more complex date manipulations if needed.","status":200,"tokens":[75,523],"elapsed":10.382690041,"cost":0.01644,"log_prob":null,"finish_reason":"stop","run_id":745337376,"sample_id":null,"_type":"aimessage"}]

code_generation/data_analysis/add_yearmonth/gpt-4-turbo-2024-04-09/conversation__InJulia__1SHOT__20240410_072033__570.json

@@ -0,0 +1 @@
+[{"content":"Act as a helpful AI assistant","variables":[],"_type":"systemmessage"},{"content":"In Julia, Given a DataFrame `df` with column `dt` representing DateTimes. Write a function `add_yearmonth` that creates a new column `ym` by extracting year and month from `dt` and concatenating them together as an integer in format: “yyyymm”.","variables":[],"_type":"usermessage"},{"content":"To implement the function `add_yearmonth` in Julia which operates on a DataFrame `df` to extract the year and month from a `DateTime` column named `dt` and creates a new column `ym` formatted as \"yyyymm\", you can follow these steps:\n\n1. Load the required package `DataFrames`.\n2. Use the `Dates` module for handling date time objects.\n3. Define the function `add_yearmonth` that calculates the \"yyyymm\" integer.\n\nHere's the complete code:\n\n```julia\nusing DataFrames\nusing Dates\n\nfunction add_yearmonth(df::DataFrame)\n # Check if the DataFrame has the 'dt' column\n if !(\"dt\" in names(df))\n error(\"DataFrame must contain a 'dt' column of DateTime objects.\")\n end\n \n # Extract year and month, format them into \"yyyymm\"\n df.ym = 10000 * year.(df.dt) + month.(df.dt)\n \n return df\nend\n```\n\n### Explanation:\n- `using DataFrames`: This loads the `DataFrames` package for DataFrame operations.\n- `using Dates`: This imports the `Dates` module which provides essential functions to manipulate dates in Julia.\n- Inside the function `add_yearmonth`:\n - It first checks if the column `dt` exists in the DataFrame. If not, it throws an error.\n - `year.(df.dt)` extracts the year part from each `DateTime` entry in the `dt` column and `month.(df.dt)` extracts the month part.\n - The result `ym` column is created by multiplying the year by 10000 and then adding the month. This way, years and months like `2023` and `4` become `202304`.\n - The function then returns the modified DataFrame with the new `ym` column.\n\n### Usage Example:\n```julia\n# Example data creation\ndf = DataFrame(dt = [DateTime(2023, 1, 1), DateTime(2023, 4, 15), DateTime(2021, 7, 23)])\n\n# Apply the function\nnew_df = add_yearmonth(df)\n\n# Display the modified DataFrame\nprintln(new_df)\n```\n\nThis example should output the original DataFrame with an added column `ym` showing integers like `202301`, `202304`, and `202107` based on the `dt` values.","status":200,"tokens":[75,496],"elapsed":9.433594334,"cost":0.01563,"log_prob":null,"finish_reason":"stop","run_id":-1333023290,"sample_id":null,"_type":"aimessage"}]

code_generation/data_analysis/add_yearmonth/gpt-4-turbo-2024-04-09/conversation__InJulia__1SHOT__20240410_072045__388.json

@@ -0,0 +1 @@
+[{"content":"Act as a helpful AI assistant","variables":[],"_type":"systemmessage"},{"content":"In Julia, Given a DataFrame `df` with column `dt` representing DateTimes. Write a function `add_yearmonth` that creates a new column `ym` by extracting year and month from `dt` and concatenating them together as an integer in format: “yyyymm”.","variables":[],"_type":"usermessage"},{"content":"To achieve the task of extracting the year and month from a `DateTime` column in a DataFrame and then concatenate these values into a new column in Julia, you can use the `DataFrames` and `Dates` packages. If these packages are not installed, first install them using Julia's package manager:\n\n```julia\nusing Pkg\nPkg.add(\"DataFrames\")\nPkg.add(\"Dates\")\n```\n\nOnce the packages are in place, you can define the function `add_yearmonth` as described. Here is how you can write it:\n\n```julia\nusing DataFrames\nusing Dates\n\nfunction add_yearmonth(df::DataFrame, col_name::String)\n # Ensure the specified column exists in the DataFrame\n if !(col_name in names(df))\n error(\"Column $col_name not found in DataFrame\")\n end\n\n # Check if the column is of DateTime type\n if !(eltype(df[!, col_name]) <: DateTime)\n error(\"Column $col_name is not of DateTime type\")\n end\n\n # Create the 'ym' column by extracting the year and month, and formatting them as yyyymm\n df.ym = map(x -> year(x) * 100 + month(x), df[!, col_name])\n\n return df\nend\n```\n\nThis code performs the following tasks:\n1. Checks if the specified column exists in the DataFrame.\n2. Verifies that the data in the column is of type `DateTime`.\n3. Uses the `map` function to apply a transformation to each `DateTime` element in the given column. It extracts the year and the month, then formats them as `yyyymm` (for example, for January 2023, it would be `202301`).\n4. Assigns the result back to a new column `ym` in the original DataFrame.\n\n### Example Usage:\n\nHere's how you could use this function:\n\n```julia\n# Import necessary packages\nusing DataFrames, Dates\n\n# Create a sample DataFrame with DateTime data\ndf = DataFrame(dt=[DateTime(2023,1,15), DateTime(2023,2,20), DateTime(2023,3,25)])\n\n# Add the 'ym' column\ndf = add_yearmonth(df, \"dt\")\n\n# Display the modified DataFrame\nprintln(df)\n```\n\nThis will output a DataFrame where the `dt` column contains the original `DateTime` values and the new `ym` column contains the `yyyymm` formatted integers.\n\nThis solution is scalable and easy to integrate into data processing pipelines dealing with time series data in Julia.","status":200,"tokens":[75,530],"elapsed":11.417722375,"cost":0.01665,"log_prob":null,"finish_reason":"stop","run_id":884663856,"sample_id":null,"_type":"aimessage"}]

code_generation/data_analysis/add_yearmonth/gpt-4-turbo-2024-04-09/conversation__InJulia__1SHOT__20240410_072117__100.json

@@ -0,0 +1 @@
+[{"content":"Act as a helpful AI assistant","variables":[],"_type":"systemmessage"},{"content":"In Julia, Given a DataFrame `df` with column `dt` representing DateTimes. Write a function `add_yearmonth` that creates a new column `ym` by extracting year and month from `dt` and concatenating them together as an integer in format: “yyyymm”.","variables":[],"_type":"usermessage"},{"content":"To accomplish the task in Julia using the DataFrame package and the Dates module to work effectively both with dates and DataFrame objects, you will need to write a function `add_yearmonth` that takes your DataFrame as an input, and then manipulates the DateTime column to extract the year and month, format them as desired, and finally add this new information back to the DataFrame as a new column.\n\nHere is a step-by-step explanation and the corresponding Julia code:\n\n1. **Ensure you have the required packages**: Make sure that you have DataFrames.jl and Dates.jl installed. If not, install these using Julia's package manager. You can install these packages by running:\n ```julia\n using Pkg\n Pkg.add(\"DataFrames\")\n ```\n\n2. **Import the required modules**:\n ```julia\n using DataFrames\n using Dates\n ```\n\n3. **Write the function `add_yearmonth`**:\n - The function will take a DataFrame expected to have a column named `dt` with DateTime type.\n - It will extract the year and month, format them, and then create a new column using the formatted values.\n\n Here's how the function could be written:\n\n ```julia\n function add_yearmonth(df::DataFrame)\n # Check if the DataFrame has the required column `dt`\n if !(:dt in names(df))\n error(\"DataFrame must contain a 'dt' column of type DateTime\")\n end\n\n # Create a new column by transforming the `dt` column\n df.ym = map(dt -> year(dt) * 100 + month(dt), df.dt)\n \n return df\n end\n ```\n\n4. **Example usage**:\n Here’s how you can test and use this function:\n\n ```julia\n # Create a sample DataFrame including DateTimes\n df = DataFrame(dt = [DateTime(2023, 1, 5), DateTime(2022, 12, 25), DateTime(2023, 7, 10)])\n\n # Apply the function\n new_df = add_yearmonth(df)\n\n # Display the modified DataFrame\n println(new_df)\n ```\n\nThis function should now accurately add a `ym` column to your DataFrame, with each entry in that column formatted as `yyyymm` from the existing `dt` DateTimes. Adjustments may be needed based on other specific requirements or DataFrame configurations you're working with.","status":200,"tokens":[75,508],"elapsed":31.819052125,"cost":0.01599,"log_prob":null,"finish_reason":"stop","run_id":-301965122,"sample_id":null,"_type":"aimessage"}]