21 Pull requests merged by 10 people

• [BFCL] Add Test Dataset to Repository

  #504 merged Jul 11, 2024

• [BFCL] Fix Possible Answer for AST Parallel and Parallel_Multiple Category

  #503 merged Jul 8, 2024

• Leaderboard Update, in sync with PR #423 & #503 (Fix Possible Answer for AST Parallel and Parallel_Multiple Category)

  #507 merged Jul 8, 2024

• [BFCL] Improved tree-sitter java, javascript installation

  #505 merged Jul 7, 2024

• [BFCL] Sanity check is now optional

  #496 merged Jul 7, 2024

• [BFCL] Update Leaderboard to reflect Nemotron-4-340b-Instruct score

  #491 merged Jul 7, 2024

• [BFCL] Update Leaderboard after adding GLM-4-9B

  #475 merged Jul 7, 2024

• [BFCL] Add Support for GLM-4-9B function calling inference

  #474 merged Jul 7, 2024

• fix some data issues in parallel/parallel multiple answers

  #423 merged Jul 7, 2024

• [BFCL] Add ability to evaluate Nemotron-4-340B-Instruct

  #489 merged Jul 5, 2024

• [GoEx] Undo Minor Bug Fix + README Minor Improvement

  #468 merged Jul 3, 2024

• Remove redundant tokens from GPT-handler

  #490 merged Jul 1, 2024

• [BFCL] Standardize Model Name Among handler_map and eval_runner_helper

  #439 merged Jul 1, 2024

• Added Agent Marketplace Blog

  #404 merged Jun 22, 2024

• Leaderboard Update, adding new model firefunction-v2-FC

  #476 merged Jun 22, 2024

• [BFCL] Add Claude 3.5 to Leaderboard

  #481 merged Jun 21, 2024

• [BFCL] Add Claude 3.5 Sonnet Function Calling Infernece Inference

  #480 merged Jun 21, 2024

• Add firefunction-v2 to the leaderboard

  #470 merged Jun 19, 2024

• [BFCL] PR#407 Evaluation Pipeline Robustness Patch

  #462 merged Jun 19, 2024

• Leaderboard Update, in sync with PR#437 (Fixes For NexusHandler)

  #472 merged Jun 19, 2024

• Fixes For NexusHandler

  #437 merged Jun 19, 2024

12 Pull requests opened by 4 people

• Blog 11: Agent Marketplace

  #483 opened Jun 22, 2024

• Added Agent Marketplace to Gorilla Repository

  #487 opened Jun 26, 2024

• [BFCL] Adds support for parallel inference and batching

  #498 opened Jul 2, 2024

• [BFCL] Standardize TEST_CATEGORY Among eval_runner.py and openfunctions_evaluation.py

  #506 opened Jul 6, 2024

• [BFCL] Overhaul apply_function_credential_config.py for Enhanced Usability

  #508 opened Jul 7, 2024

• [BFCL] Update BFCL Manual Blog to Reflect PR #407

  #509 opened Jul 7, 2024

• Make BFCL User-Friendly and Easy to Extend

  #510 opened Jul 7, 2024

• [BFCL] Support Category-Specific Generation for OSS Model, Remove eval_data_compilation Step

  #512 opened Jul 8, 2024

• [BFCL] Specify package version in requirements.txt

  #515 opened Jul 8, 2024

• [BFCL] Fix Double-Casting Issue in model_handler for Java and JS category.

  #516 opened Jul 8, 2024

• [BFCL] Improve Warning Message when Aggregating Results

  #517 opened Jul 9, 2024

• [BFCL] Fix Dataset Issue for executable_parallel_multiple Category

  #522 opened Jul 10, 2024

10 Issues closed by 4 people

• distutils.errors.CompileError: command '/usr/bin/cc' failed with exit code 1

  #520 closed Jul 9, 2024

• [Apibench] No module named 'tree_sitter_java'

  #514 closed Jul 9, 2024

• Question about AST evaluation for Java and JavaScript

  #477 closed Jul 7, 2024

• [BFCL] Sanity check should be optional and by default off

  #486 closed Jul 7, 2024

• LeaderBoard data generation

  #499 closed Jul 5, 2024

• Java/Javascript Scores

  #495 closed Jul 3, 2024

• [bug] OpenFunctions-v2: how to continue conversation?

  #488 closed Jun 29, 2024

• [bug] OpenFunctions-v2: <HTTP code 502>

  #467 closed Jun 24, 2024

• [bug] OpenFunctions-v2: <Issue>

  #466 closed Jun 24, 2024

• Rapid API error (Yahoo Finance, https://rapidapi.com/sparior/api/yahoo-finance15) is inaccessible

  #456 closed Jun 12, 2024

14 Issues opened by 10 people

• Test data error in executable parallel multiple function

  #519 opened Jul 9, 2024

• Evaluation using vLLM and other tools

  #518 opened Jul 9, 2024

• Questions about the evaluation criteria.

  #513 opened Jul 8, 2024

• Single Source of Truth

  #511 opened Jul 8, 2024

• Clarify Documentation About Running The Benchmark

  #502 opened Jul 6, 2024

• BFCL setup instruction is very difficult to follow

  #501 opened Jul 6, 2024

• Set Model Temperature to 0 for Consistent Leaderboard Results

  #500 opened Jul 5, 2024

• Question about AST evaluation for Java

  #494 opened Jul 1, 2024

• [BFCL] Inconsistency in leaderboard scores

  #493 opened Jul 1, 2024

• [BFCL] Get rid of legacy naming convention for LLM generated files

  #485 opened Jun 25, 2024

• [Apibench] Resume interrupted LLM generations from last generation

  #484 opened Jun 24, 2024

• [RAFT] Publish Pypi package with raft, eval and format scripts

  #478 opened Jun 19, 2024

• Data issue

  #471 opened Jun 14, 2024

• When [Evaluate the Response with AST tree matching]: TypeError: __init__() takes exactly 1 argument (2 given)

  #469 opened Jun 12, 2024

6 Unresolved conversations

Sometimes conversations happen on old items that aren’t yet closed. Here is a list of all the Issues and Pull Requests with unresolved conversations.

• [feature] Add multi-turn conversational function calling category for benchmarking

  #442 commented on Jun 17, 2024 • 0 new comments

• Revamp Landing README

  #463 commented on Jun 26, 2024 • 0 new comments

• Java/Javascript split questions

  #424 commented on Jul 1, 2024 • 0 new comments

• Frontend Deployment for Add-API

  #269 commented on Jun 19, 2024 • 0 new comments

• FC alignment

  #413 commented on Jun 23, 2024 • 0 new comments

• Add Sticky Effect for the First Three Columns on Leaderboard

  #431 commented on Jun 19, 2024 • 0 new comments