Skip to content

Evaluation of code generation, with sequential trials, using PaLM 2

License

Notifications You must be signed in to change notification settings

barun-saha/CodePrompt

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

4 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

CodePrompt

A collection of prompts to investigate code generation using PaLM 2.

The dataset consists of 30 coding problems in Python. These range from code generation to completion and troubleshooting errors. A summary of these problems is available in the accompanying spreadsheet.

The CodePrompt dataset takes into consideration the sequential nature of trial and error that is often required with Large Language Models (LLMs) in order to obtain the desired output (or the output in a desired format). For example, some problems can be solved at the first attempt. Some might need a second attempt by refining the input prompt, a third attempt after refining further refining the input prompt, and so on. Accordingly, three maximum attempts are considered here. In case no correct output is generated even after the third attempt, a comment is provided in the spreadsheet explaining the reason.

If you are using CodePrompt, please cite the following paper:

B. K. Saha, “Generative Artificial Intelligence for Industry: Opportunities, Challenges, and Impact” in Proceedings of 2024 International Conference on Artificial Intelligence in Information and Communication (ICAIIC), IEEE, Feb. 2024.

Releases

No releases published

Packages

No packages published

Languages