-
Notifications
You must be signed in to change notification settings - Fork 2.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
-e in pip install -e . fails (linux and wsl2) [Solved: Update pip] #269
Comments
Hello, I had this issue as well. But when I upgraded pip to the latest version, it worked for me. |
I guess it's the problem of your enviorment |
|
Thank you so much for this. Was having this issue on Ubuntu Server 22.04 |
inwaves
added a commit
to inwaves/evals
that referenced
this issue
Mar 19, 2024
commit af58ab97de097ab65cf8c5bbf4b58c1abf645f6c Author: Andrei Alexandru <[email protected]> Date: Tue Mar 19 03:14:40 2024 +0000 Remove Ainu dataset from skill acquisition (#345) * Remove Ainu dataset * Remove Ainu dataset commit 5317164a5091db464772da76e18c5806523a7cba Author: Oliver Jaffe <[email protected]> Date: Fri Mar 15 14:37:48 2024 +0000 Multithreading for GeminiSolver (#330) * GeminiSolver supports multithreading * add documentation for glm client setup * better doc * only share client between solver copies * implement backoff * add model version property commit ac9024921855cff4e698718d10de11b91cab19b4 Author: Giulio Starace <[email protected]> Date: Fri Mar 15 15:24:32 2024 +0100 AnthropicSolver (#331) * add anthropic to pyproject toml * anthropic solver skeleton * name property * mvp; need to fix backoff/rate limiting; need to test cot * forgot to commit registry entry * implement backoff * implement alternating roles to support CoT * fill in the rest of the yaml with remaining anthropic models * log usage * make it a static method * anthropic solver pytests * update docs; implement model_version * dont use abbreviation Co-authored-by: Oliver Jaffe <[email protected]> * let _solve accept kwargs * include haiku (came out today) * switch ordering of haiku and sonnet * reqs handled by pyproject.toml --------- Co-authored-by: Oliver Jaffe <[email protected]> commit 209fa98a10604b3d7645ab64dec148cb167abc16 Author: Dane <[email protected]> Date: Sat Mar 16 00:25:59 2024 +1100 Refactor MLAB v2 (#340) commit 081d74af1701fef72c8c0ad39e847e05a168289f Author: Oliver Jaffe <[email protected]> Date: Fri Mar 15 10:26:42 2024 +0000 correct eval names for error recovery (#341) commit 66684a555b0abe51d1be3c09cf33278bf67a7f76 Author: Dane <[email protected]> Date: Fri Mar 15 19:56:27 2024 +1100 Update MLAB LICENSE.md (#338) commit f2a900a6aa6c356305817fb0282073b97deaa291 Author: Chan Jun Shern <[email protected]> Date: Fri Mar 15 06:33:07 2024 +0000 Final cleanup for release (#339) * Remove redundant httpx logging settings * Minor bugged tools readme * minor cleanup and precommit icrl * Cleanup fdeduct todos commit 7f5f8c6e44b22d11f51ccc506ebee0bd161b8d3d Author: James Aung <[email protected]> Date: Thu Mar 14 14:05:49 2024 -0700 Update contribution statements in READMEs (#337) commit cadff08108a7c81c15561d968fb8ebc3ab7832c5 Author: James Aung <[email protected]> Date: Thu Mar 14 05:40:14 2024 -0700 LICENSE files for Error Recover and 20 Questions (#335) * 20Q and Error Recover dataset Licenses * Create LICENSE files for 20Q and ER datasets commit b7cffc0cf06b2cd3fd42a9491f0f6fbffa15c242 Author: Oliver Jaffe <[email protected]> Date: Thu Mar 14 12:11:17 2024 +0000 skip test (#334) commit ffd004e65c7b7dda4f726916dc1fd453d7368347 Author: Dane <[email protected]> Date: Thu Mar 14 19:54:07 2024 +1100 Add licenses to MLAB v2 (#333) * Create LICENSE * Update LICENSE.md commit 5815222c56f8e0b186bbe8ba69f189e704db17e6 Author: Dane <[email protected]> Date: Thu Mar 14 19:42:26 2024 +1100 Update LICENSE.md (#332) commit 3a00e556ea1a1e18eb7063321e14d8534bf0e48d Author: Giulio Starace <[email protected]> Date: Thu Mar 14 08:53:16 2024 +0100 Coherence - final PR (#325) * get rid of .dev alias * ast precommit checks+fixes * tts precommit checks+fixes commit c71db16e4fafa93d44ec6dec58a24cbd5ad82259 Author: Dane <[email protected]> Date: Thu Mar 14 02:06:33 2024 +1100 Update MLAB v2 scripts (#323) commit 7399673ad004ca32a2bf668df1d7a3cdf11df5fa Author: Giulio Starace <[email protected]> Date: Wed Mar 13 14:27:21 2024 +0100 wordnet license (#329) commit d3bf8c96c5aca7a3733dcee985de7f1b77ba32d4 Author: Giulio Starace <[email protected]> Date: Wed Mar 13 12:53:48 2024 +0100 AST - Use wordnet corpus rather than brown corpus (#328) * wordnet, not brown * fix path * wordnet words, not brown words * wordnet corpus, not brown commit 024ee14e7d06bf8f1d3f5a83f6e0ce84794e4992 Author: Ian McKenzie <[email protected]> Date: Tue Mar 12 23:17:23 2024 -0700 Multi-Step Web Tasks (formerly Bucket of Tasks) (#49) * add partial webarena * very basic task runnable (for real) * replace BoTSolver with Solver * change TaskState, change log, remove some redundant code/files * change logging debug to info * change to using local rather than global docker_client * delete unnecessary files * set up skeleton of BashBrowserEnv * add BashBrowser auxiliary classes * add more BashBrowser stuff, and fix running multiple tasks in sequence * simple task that uses bash and browser working! * setup and teardown code for simple-web working * abstract more container loading logic, downloading is untested * fix downloading, improve container setup * move Session class to separate file * add (untested) setups for webarena containers * basic setup working for all environments * remove accidentally committed scratch file * add gpt-3.5 solvers * refactor container setup, add networking * register bash with session, set up networking * remove input statement, add more waiting * move bash setup into Session and enable building from Dockerfile * add bugged internet disabling * disabling internet access seems to work * delete unused agent code * add try..except around Session setup * fix stopping condition to include early stopping * set up BoTTaskState such that default Solvers can try the task * add the three easy tasks * slightly edit the agent prompt to see if it helps * fix easy Python task! * change to gpt-4-32k * fix BashObservation to have data property, not method * more prompt tuning * add way to run environments in repro * start adding medium/hard tasks (not complete) * add homepage docker container (first draft) * remove unused apps from homepage * add homepage to Session * add medium tasks, fix evaluator * add individual tasks for testing * fix ProgramHTML evaluation * replace 'match' with 'elif's * fixes to make medium and hard tasks run * clean up datasets and yaml * fix '|' mistakes in match-statement-replacement elifs * add gitlab url fix * remove unused field from json * add reproducibility run_experiments.sh script * solvers and 3.9 compat: replacing '| None' with 'Optional' and 'match' with 'elif' * more 3.9 compat changes * add longer timeout to ready check (for simple-web) * small task fixes * some logging tweaks * some prompt tweaks * change to using StrongSolver in run_experiments.sh * re: Dane's review, add timeout in constants, fix elif * update gitlab task to say 'main' * update prompt to emphasize homepage * remove redundant network from bash env * fix homepage links * save final report as dictionaries with task_ids * add some scratch code to run the bash container too * fix task 7 to use different repo * add sleep and empty check to avoid issues with browser env * add hack to prevent 'goto' accessing the internet * use 0613 checkpoints * add homepage to all tasks * use model context length to choose observation history * change to using 0613 snapshots * change message fitting to cut long observations * have Session log errors that cause it to shut down * switch default action splitter to single backtick * add first draft of README * add explicit check for chat model * Revert "switch default action splitter to single backtick" This reverts commit 813d832b26ba3aefebc6144a12812adf0f9968f0. * add action splitter to previous actions in prompt * reduce chars per token estimate from 3.2 to 3 * update wikipedia task to use Lorentz rather than Croatian election * change logging file extensions to match other evals * update StrongSolver to use tiktoken to fit context length * start on data parsing for plots * first version of plotting * change wikipedia task to not be answerable by gpt-4 * modify prompts based on jun's feedback * remove vscode file * update import and remove redundant todo * make output dir automatically in make_plots, add to run_experiments.sh * tidy up plot, add descriptive labels * changes from Dane/Jun comments * add contribution statement to README * change Eval -> SolverEval * add cleanup script to reproducibility * remove old task file * remove 'BoTSolverResult' * remove unused 'browser_early_stop' fn * move requirements into pyproject.toml and add setup instructions * improve setup instructions in README and improve error handling in session.py * initial version of playwright-flask app * two options: exec or define all functions * have a draft of basic structure of client and server code down * remove unused 'run_function' methods * debugged some issues with return values * first draft of dockerizing api (untested) * more incremental progress on flask-playwright api * incremental: remove iptables, debug more commands * change cleanup to prune networks, and add wikipedia to run_environments * refactor to use 'Forwarder's everywhere, and change how ports are used * add separation between client urls and server urls * add flask-playwright to CLEANUP * end-to-end run of task 1 successful * add slightly better logging to failed actions * remove 'bridge' network, containers only accessible via 'bucket-of-tasks_network' now * fix task urls * update strong solver prompt (fix typo, add https://homepage.com) * change gitlab container to use 'https://gitlab.com' as its url * add better resetting, try to debug Execution context error * just wait after navigation commands * fix bugs with retrying and base urls inside containers * fix issue with quoting in page.evaluate and fix bug with retry logging * remove redundant url fixing method and continue after invalid url in goto * better error logging in evaluators.py * change urls on homepage to be docker internal * change to using homepage as start url for all tasks * update debugging script * remove hardcoded hack to ensure urls start with 127.0.0.1 * revert change to gitlab internal url (should be gitlab:8023 for git clone) * change curl command logging info -> debug to reduce spam * move BoT reproducibility to elsuite * fix output dir in run_experiments * remove messy debugging scripts * remove linux restriction now that we use playwright container * remove exposed ports from container setup * change cache dir to be more portable * rename bucket-of-tasks to multistep-web-tasks * rename bucket of tasks in README * change default task to simple-web for CI, fix minor bugs * Remove commented-out code * Add spacing for docstrings --------- Co-authored-by: Dane <[email protected]> commit c4c9517bb9ef2eee705a2d1ad00b9a805f2ab33c Author: Giulio Starace <[email protected]> Date: Mon Mar 11 12:48:51 2024 +0100 Coherence - Readmes + minor cleanup (#316) commit ebf453c16aeace5d9884aa809aae3c0ebc5f3a99 Author: Oliver Jaffe <[email protected]> Date: Mon Mar 11 10:37:37 2024 +0000 Bugged Tools - minor fixes (#315) * don't use judge with DummySolver * basic error handling around tools commit b1c5cc0339d66202efcf4a1868f07b96025fa6aa Author: Chan Jun Shern <[email protected]> Date: Mon Mar 11 08:50:57 2024 +0000 Function Deduction plotting code (#305) * Redo plotting code * Make colors less jarring * Fix missing text annotations bug * Update evals/elsuite/function_deduction/scripts/make_plots.py Co-authored-by: James Aung <[email protected]> * Update evals/elsuite/function_deduction/scripts/make_plots.py Co-authored-by: James Aung <[email protected]> --------- Co-authored-by: James Aung <[email protected]> commit ce40f1eafb653fa3da30c003f64b8132cbd0bb8b Author: Oliver Jaffe <[email protected]> Date: Mon Mar 11 08:27:30 2024 +0000 don't attempt to sum None (#314) commit c92857de8df82cc74861b493954c28dd239ab0ef Author: Dane <[email protected]> Date: Mon Mar 11 19:20:18 2024 +1100 Standardize terminology in MLAB v2 baselines (#321) * Update baselines to report "return" instead of "reward" * Update bipedal walker baseline * Update random sampling seed in naive baselines commit e47ce920b4290209bf77216da1b5532eae788fed Merge: e3b7360b 7efc3f65 Author: Chan Jun Shern <[email protected]> Date: Mon Mar 11 08:17:48 2024 +0000 Merge pull request #322 from openai/jun/change-steg-datasets Drop two datasets from steganography commit e3b7360b13de77c9b901d3e33d2eff939ffcac7e Author: James Aung <[email protected]> Date: Mon Mar 11 01:02:30 2024 -0700 Add InContext RL READMEs (#318) * Create README * Add ICRL to main README * Add Dataset section --------- Co-authored-by: Chan Jun Shern <[email protected]> commit 40b5f144325c75dfbf3087692348a3e214490129 Merge: 7e85428d 030b2324 Author: Chan Jun Shern <[email protected]> Date: Mon Mar 11 06:15:55 2024 +0000 Merge pull request #320 from openai/jun/sync-oss-20240311 Jun/sync oss 20240311 commit 7efc3f6573182d039cd7fa5de729c195e26e8cbb Author: Chan Jun Shern <[email protected]> Date: Mon Mar 11 12:50:58 2024 +0800 Drop two datasets from steganography commit 030b2324798b81a043c0a56fa0b75e33317a72bf Merge: 7e85428d 82ec660e Author: Chan Jun Shern <[email protected]> Date: Mon Mar 11 12:12:22 2024 +0800 Merge remote-tracking branch 'public/main' into jun/sync-oss-20240311 commit 7e85428db2a1587f9a824be23543be32f78cd13e Author: James Aung <[email protected]> Date: Sun Mar 10 20:20:48 2024 -0700 In-Context RL plotting (#280) * init template scripts * plotting code * correct plots * plotting code improvement * Got plotting working in notebook * Updated line styles for baselines * Change opacity of lines * Un-messup merge * Update anti-cot solver for 4-turbo and the new 3.5 * Run experiments * Enable printing of each command before execution in run_experiments.sh script * New plotting code * Plotting code working * Delete old files * Add final average reward as a metric * Fix threading issue? * fix explanations arg name * change what solver we use to be generation/direct * no longer run sequentially thanks to threadding fix * Change qlearning baseline to train for max steps instead of max episodes * fix too many messages * new plotting code * qlearning for 1m steps * Add loop to run experiments multiple times * pretty names * Add filter script for log file processing * adjust fig size and add labels * New custom map for FrozenLake * Change max_steps to 200 * add evaluation function * simplify plotting 😌 * Fix saving plot to correct directory * per-env window sized and prettify * annotate lines with final values * invalid response rate plots * Update qlearning_baseline.ipynb * add labels * Catch broader class of GoogleAPIError * Widen catch for response.text errors * adjust invalid action message * new episode reward metrics * del plotting playground * Fix _calculate_episode_rewards method signature * Add handler for response.parts as well * cleanup plotting * update qlearning baselien for correct custom map * add short variant * max rolling avg in json * remove filter notebook * fix max reward of windows * Fix final settings for experiment script * Clean up prettifying functions to be dicts instead * clenn up episode rewards calculation in eval.py --------- Co-authored-by: Chan Jun Shern <[email protected]> commit 25aa67e6babcb32dfd6ac92ab1e87cb9387f321b Author: Dane <[email protected]> Date: Sat Mar 9 20:48:28 2024 +1100 Fix spelling mistake in HRMLAB prompts (#319) commit caa46568eafbc6e7fc7d7ccd662e70d141fe02eb Author: Oliver Jaffe <[email protected]> Date: Fri Mar 8 16:28:12 2024 +0000 prevent division by zero (#317) commit d872aeb3bb326790078dc24fe9498e25ee802926 Author: Oliver Jaffe <[email protected]> Date: Fri Mar 8 11:20:08 2024 +0000 CDTA - readme (#313) * add eval README * compute headline metric * add metrics + description to eval yaml. Add shorter * add shorter version of eval * fix dataset dupliation bug * readme updates * point users towards prompts in readme * comment on custom chess solvers * add CDTA to root README * Update evals/registry/evals/cant_do_that_anymore.yaml Co-authored-by: Giulio Starace <[email protected]> * replace subsequent -> special * Update evals/elsuite/cant_do_that_anymore/README.md Co-authored-by: Giulio Starace <[email protected]> * mention more explicitly that each solver has a unique dataset * Update evals/elsuite/cant_do_that_anymore/README.md Co-authored-by: Giulio Starace <[email protected]> * include both input and output tokens * include all metrics in eval yaml --------- Co-authored-by: Giulio Starace <[email protected]> commit 90f0bc6f95985243cb31107cbee5747772ea182a Author: Dane <[email protected]> Date: Fri Mar 8 21:21:07 2024 +1100 Add OpenAI Gym environments to HR-MLAB (#242) * Merge commit with main * Update plotting code * Update default max steps to be `30` * Add gym to HR-MLAB requirements * Tidy up vectorization's `train.py` * Add the BipedalWalker-v3 environment * Add the CartPole-v1 environment * Add partially-implemented llama inference task * Update Cart Pole baselines * Update baselines and scoring for BipedalWalker * Update Cart Pole baseline and grading * Update bipedal walker baselines and grading * Add the inverted pendulum env * Fix human baseline for inverted pendulum * Fix bipedal walker baselines * Add pusher environment * Add task descriptions and update yaml * Add the `Ant-v4` env * Add `Ant-v4` data and yaml * Add `Pong-ramDeterministic-v4` env * Make Ant normalization fn clearer * Update Ant grading * Add `Humanoid-v4` env * Update Cart Pole grading * Update inverted pendulum grading * Updating `Pusher-v4` grading * Update Pong human baseline and grading * Update yaml and add humanoid jsonl * Update humanoid grading * Update humanoid human baseline and grading * Update the bipedal walker grading script * Update bipedal walker grading docstring * Update `Cart Pole` time limit and baselines * Update time limits for inverted pendulum * Update bipedal walker max time limit * Update yaml file * Update `Ant-v4` environment * Update `BipedalWalker-v3` env * Update `CartPole-v1` env * Update `Humanoid-v4` env * Update `InvertedPendulum-v4` env * Update `PongNoFrameskip-v4` env * Update `Pusher-v4` env * Update .gitignore * Add time limit to attempt * Update the `Ant-v4` env * Update the `Humanoid-v3` env * Refactor get_baseline_score function to allow for additional files and saving checkpoints * Refactor human baseline script and add checkpoint file * Remove unnecessary code for file copying * Add cache decorator to score calculation functions * Add cache decorator to score calculation functions * Update CartPole human baseline * Cache baselines for humanoid env * Cache baselines for inverted pendulum env * Cache baselines for pong * Cache baselines for pusher * Update experiment scripts * Update timeout error message in environment.py * Create logic to stop eval on out of context error * Update .gitignore * Add time and steps remaining reminder for model * Make normalization functions linear * Handle context length exceeded in solver * Update README * Merge pyproject.toml * Add `max_time` parameter to v1 jsonl files * Remove default max steps etc. and improve instructions * Remove `shell=True` from file execution fn * Remove video recorder from baselines * Add script to calculate token estimates * Apply hooks * Refactor error message for invalid action input * Update task descriptions for llama inference and vectorization * Refactor solver class name to SimpleActionAgent * Update assertion error message in baseline solver * Remove commented-out code in autoeval.py * Update run_experiments.py * Update Bipedal Walker task description * Simplify token consumption estimate * Update README.md * Handle edge case for unknown completion fn * Change baseline solver to use `OpenAISolver` instead of `OpenAIChatCompletionFn` * Re-add files with Git LFS tracking commit 2e0676e14616895baa5456a72c822476feef906a Author: Chan Jun Shern <[email protected]> Date: Fri Mar 8 09:55:40 2024 +0000 Final plots for error rec (#312) commit 1fa967fa8d470f88da0ea780666c250069a5a717 Author: Ian McKenzie <[email protected]> Date: Fri Mar 8 01:48:48 2024 -0800 [Skill Acquisition] Changes from rerun (#300) * Fix copy bug for subclasses * Add retries with backoff * Add task_description to record_sampling * Add SkillAcqAssistantsSolver along with yaml entries * Log token usage in final report * reduce yaml + run exps script to just solvers used * changes for rerun * add back some solvers * remove my absolute path * Update plotting script * Remove commented out code --------- Co-authored-by: Chan Jun Shern <[email protected]> commit e775967ecd1d9c639593d78d2ed5f612fafb1a5a Author: James Aung <[email protected]> Date: Thu Mar 7 07:58:52 2024 -0800 error recovery README commit 752432e3112f2eb387fa6ff6213ee651c5c193f6 Author: James Aung <[email protected]> Date: Thu Mar 7 07:58:23 2024 -0800 Update README.md (#309) commit b9938df47d4ddbc3baffbef859e0d18f5cdd5e29 Author: Oliver Jaffe <[email protected]> Date: Thu Mar 7 15:53:35 2024 +0000 Mode Collapse - Variants + Polish (#307) * add option to create control dataset * rename dataset creation script * various readability + documentation improvements * log how many games have been examined during dataset creation process * only update tqdm bar + dataset if more than one move is found * remove unnecessary any_previous_move_found var * add option to save dataset throughout creation * remove unused n_threads arg for dataset_creation.py * remove unused recorder * skeleton structure for creating diagonal dataset * logic for finding diagonal moves * add support for diagonal experiment * remove unused plot. Render headline metrics * add plot for diagonal variant. Plot performance vs. num previous bishop moves * update progress bar with correct num. new examples * fixed bug where dataset would contain duplicate examples * run reproducability multiple times to compute SE * reproducability script to run diagonal variant * wrap single move in main+control datasets in list * seed changes on each different eval run * combine run_experiments scripts * remove unnecessary sort * add chess to pyproject.toml. Also correct typing * rename eval dir mode_collapse -> cant_do_that_anymore * rename mode collapse -> cant do that anymore * add diagonal dataset. * add required base solvers * up reproducability n_samples to 1000 * remove marker from plots. Provide correct path for saved figure * update datasets to use new keys * use default dataset for DummySolver commit 7f5f2a61d530485892a1e9a006e82006a3c3a7a3 Author: Giulio Starace <[email protected]> Date: Thu Mar 7 13:36:28 2024 +0100 Coherence reproducibility (#261) * repro script for already_said_that * make it exectuable * solvers on outer loop * use aliases rather than specific snapshot * autoformat * missing gpt-4-base solver * temporarily suspend cot solvers * switch to 10 threads * run_experiments script for track the stat * temporarily disable gpt-4-base * .log; temporarily skip gpt-4-base * specify seed * additional helper method for specs * distractorless baseline * formatting * script for making track_the_stat plots * already_said_that plotting * increase thread count to 100 * enable gpt-4-base * explicit gpt-4-base was missing * get gpt-4-base too! * 50 threads; 100 causes hanging issues * indefinite (or long) hanging iwth more than 10 threads. sorry do gpt-4 first * use first letter rather than find-letter * clarify task even more * switch to first letter * all seeds of one solver before moving to the next * first letters (plural) * first letters plural * fix label; make results_dict init more general * use a global var for MODELS; correct first-letters to plural * fix legends * rename ideal to max * divide up plots and json * update cot solver names; only run gpt-4-turbo-preview cot by default * prepare plotting for gpt-4-base and cot 4 turbo * n_samples to match what ive been running (250 rather than 500) and reproducibility * make distractorless a bar * implement token counting * token counts in track the stat * handle CoT models * styling * run exps wrapping up * integrate gemini * tts explicit state for gemini and together * allow specifying the role of the explicit state message * integrate gemini in tts run exps * random baseline * Catch broader class of GoogleAPIError * Widen catch for response.text errors * integrate random baseline * remove extra comment * add direct solvers for together models * integrate together models in tts run exps * fix shortsight in comment * Add handler for response.parts as well * move gemini to bottom * handle new models * integrate random baseline * move main to the bottom * all stats in json stats * handle single seeds; third party models * plot random and human baseline * adjust plot size * switch to 0.05 * snapshot name * move main to bottom * dont hardcode num repeats * gemini pro 1.0 * better labels * turns, not steps * annotate n_samples as Optional * contract array definition/declaration * add diagnostic echo statements in loop * add explanatory comment * no longer need stat to legend loc * dont hardcode the length of the samples to 500 --------- Co-authored-by: Chan Jun Shern <[email protected]> commit 1b85e50f1bfdc76ce8258054700c3d17bba473ba Author: Giulio Starace <[email protected]> Date: Thu Mar 7 10:40:20 2024 +0100 Id vars - support for mixtral/gemini/llama in plots (#304) * autoformat * integrate third party solvers * clarify that third party is corrset only * tell users why were skipping commit 0acb25cc4a79b5a22e4e674030a1cee6dd8c7ecf Author: Oliver Jaffe <[email protected]> Date: Thu Mar 7 09:20:45 2024 +0000 Mode Collapse - Automatically Create Dataset (#294) * add model_version property to solvers * solvers that aren't nested now have str model_version * rename model -> solver. Initialise solver before calling get_solver_predictions * optionally create dataset for evaluated solver if doesn't exist * if solver isn't nested solver, clone to force temp=0 * special_move_creation now doesn't compute solver predictions * update datasets * update args for reproducability script * move model prediction logic outside of special_move_creation.py * don't attempt to remake solver with temp=0 * update default eval args to match standard setup * use dummy recorder for dataset creation * add special moves dataset to git lfs * move funcs in mode_collapse/scripts/utils.py -> mode_collapse/utils.py * add warning when generating dataset * up num. samples to 1000 to match default setup commit 751004bf500cca60ee581f1e7fee98579abe4289 Author: Andrei Alexandru <[email protected]> Date: Thu Mar 7 09:09:46 2024 +0000 20 questions: add readme, final update to plotting code (#306) * WIP plotting and repro code * Add dataset generation script and generate datasets * Remove plurals from lexicon dataset * Update repro & plotting code, add explicit max_questions arg to specs * Uncomment other solvers * Remove sequential run, uncomment standard * Remove dataset creation script, other datasets, update YAML and repro * Add final dataset, accept lowercase guesses for proper nouns * Fix word difficulty constant in plotting code * Include variant as plot title * Less verbose plot titles, update prompt, temperature value * Add baseline, labels * Add readme, final updates to plotting code * uppercase 'they' in prompt * Swap names in contrb statement commit 2ca19c356ec79a9f24cf0ed37c041e97512bbc83 Author: Oliver Jaffe <[email protected]> Date: Thu Mar 7 08:56:42 2024 +0000 Add model_version to solvers (#288) * add model_version property to solvers * solvers that aren't nested now have str model_version commit 3df5e01870829d641aad22edf7771d6d34bf1377 Author: Oliver Jaffe <[email protected]> Date: Wed Mar 6 13:51:09 2024 +0000 Mode Collapse - Tweaks & Violations (#293) * update task description. Include more information on notation * raise error if notation parser is given incorrect input * construct controller for all solvers * measure and log violations * log std of previous move length * remove random solver * tidy incorrect notation handling * split variable init * tidy violation metric calculations * better naming of specific violation * move get_binary_avg to mode_collapse/utils.py, add typing commit cff5aba866cc85fd1e541de2afcd1be2594e05b7 Author: Chan Jun Shern <[email protected]> Date: Wed Mar 6 03:11:04 2024 +0000 AssistantsSolver for SkillAcq (#291) * Fix copy bug for subclasses * Add retries with backoff * Add task_description to record_sampling * Add SkillAcqAssistantsSolver along with yaml entries * Place files under current_state['files'] * Include files on every message in thread instead of only one; remove redundant self.all_uploaded_files variable * Expand SkillAcqAssistantsSolver to SkillAcquisitionAssistantsSolver commit 037b1765e8c795d332261b022964e0834d0f10a5 Author: Chan Jun Shern <[email protected]> Date: Wed Mar 6 03:10:28 2024 +0000 Log token usage in final report (#298) * Log token usage in final report * Print comma-delimited numbers commit 827f906736be5f2f55f43a2f0a8320e7f6ec4f4a Author: Chan Jun Shern <[email protected]> Date: Tue Mar 5 03:03:27 2024 +0000 Jun/gemini bugfixes (#302) * Catch broader class of GoogleAPIError * Widen catch for response.text errors * Add handler for response.parts as well commit 038e43530f7dbf1c154a48952d3af6198f0144b1 Author: Giulio Starace <[email protected]> Date: Mon Mar 4 12:06:01 2024 +0100 Coherence - solver config for third party models (gemini/together) (#301) * tts explicit state for gemini and together * allow specifying the role of the explicit state message commit 60be685819f24e44bab014fde330ecefc6c492b9 Author: Giulio Starace <[email protected]> Date: Fri Mar 1 17:39:41 2024 +0100 Coherence - Random baseline solvers (#266) * random baseline solver for track the stat * random baseline solver for already said that * add registry=None to inits so that they dont crash see #267 * more consistent rounding commit 9b6ee76d463683cd8974f34e4f03b6f866567d2e Author: Ian McKenzie <[email protected]> Date: Fri Mar 1 01:26:43 2024 -0800 [Solvers] Add 'stop' sequences to base model calls (#295) * use msg seps as stop seqs for base model api * log warning if too many stop sequences are used commit aa7431deb3b493d72626699c3fda5ae30f2e23d0 Author: Ian McKenzie <[email protected]> Date: Thu Feb 29 14:43:04 2024 -0800 [Solvers] Add open-source models from together.ai (#284) * initial version of together.ai solver * modify messages in TogetherSolver to fit format * add mixtral and 70b * add scripts for running on os models * add script for running evals over weekend * add sleep between evals * fix for loop in bugged_tools script * use 10 threads * make skill acq scripts executable * handle together.ai context length errors * add cot and custom os solvers * switch to using cot solvers * add run_os_experiments for error recovery * change run_os_experiments to executable * fix custom solvers to use correct model names * use n_repeat=1 to match gemini * remove print statement * fix bugged_tools log naming * fix skill_acquisition log naming * os plotting changes for fdeduct and ivariables * add os/gemini models to error recovery plots * revert os plotting/experiment scripts * add default api base url * remove unused completion_fn config for together * add optional message merging and tests commit 0e8b05d18b8bd737a39c48107c606b8c0dee91fb Author: Chan Jun Shern <[email protected]> Date: Thu Feb 29 19:56:49 2024 +0000 [Solvers] Add GeminiSolver (#263) * Add initial working prototype for GeminiSolver * Add basic tests, add solvers/gemini.yaml * Add google-generativeai to pyproject.toml * Relax safety settings; handle API errors gracefully * Remove redundancy in model name * Revert to resp.text instead of longer version * Make test case less ambiguous * Update docstrings * Add postprocessors and CoT solver * Log messages in non-google format for logviz compatibility * Explicitly require EVALS_SEQUENTIAL while we haven't figured out threading * Catch known error about while we haven't figured it out * Drop postprocs that were causing issues with CoT * Fix failing test * Register gemini for fdeduct and skillacq eval-specific solvers * point to github issue for gemini threading todo --------- Co-authored-by: Ian <[email protected]> commit 0912197f18b0d4a27a2fb1d1764f5e52e2cfbccc Author: Chan Jun Shern <[email protected]> Date: Thu Feb 29 18:17:04 2024 +0000 [Skill Acquisition] Jun + Ian bug fixes (#289) * Fix bug in get_average_bleu_score * Compute acc for non-translation q's only * Log more metrics * Use full paths for files in current_state * Fix wrong key name * Log number of translation and non-translation samples in final report * fix typo in 'wrong_section' prompt * stop removing quotes from model outputs --------- Co-authored-by: Ian McKenzie <[email protected]> commit 19aa0f48759fb6881486b8670dcdd2edb445f740 Author: Chan Jun Shern <[email protected]> Date: Mon Feb 26 15:19:50 2024 +0000 Add handler for oai max messages exceeded (#290) commit 5706f315c94748e41e48ea02f71ce3da3b5657a7 Author: Oliver Jaffe <[email protected]> Date: Fri Feb 23 09:42:49 2024 +0000 Mode Collapse - reproducibility (#276) * select dataset depending on evaluated model * add random solver * record avg. number of previous moves in dataset * reproducability code for running and plotting main experiment * dataset creation follows models defined in eval.py * tqdm has correct target length * fix plotting bug, check if value is True correctly * sample dataset consistently between runs * remove preprocessors * add strip to solver ouputs since postprocessors no longer used * only create controller if necessary * pass current state as None if no legal moves are passed * rename plot_experiments.py -> make_plots.py * add TODOs for remaining work * use snapshot for gpt-4-turbo commit f6f6a765477eda6391083cd92fa16a0f42797000 Author: Andrei Alexandru <[email protected]> Date: Fri Feb 23 10:01:00 2024 +0200 20 questions: datasets + repro/plotting code (#258) * WIP plotting and repro code * Add dataset generation script and generate datasets * Remove plurals from lexicon dataset * Update repro & plotting code, add explicit max_questions arg to specs * Uncomment other solvers * Remove sequential run, uncomment standard * Remove dataset creation script, other datasets, update YAML and repro * Add final dataset, accept lowercase guesses for proper nouns * Fix word difficulty constant in plotting code * Include variant as plot title commit c6f24dca6788185af403834ee0b1a86008518064 Author: James Aung <[email protected]> Date: Fri Feb 23 07:50:07 2024 +0000 String explanations and frozenlake variant for InContext RL (#283) * refactor main loop to use a single for loop * Change logging severity level * Switch out keys for an explanations string * Add FrozenLake variant with custom map * Update evals/elsuite/incontext_rl/defaults.py * Update use_explanations variable name commit 6f83b69be681b3fd7a79fd8ac7c8caf9a325c2b5 Author: Ian McKenzie <[email protected]> Date: Thu Feb 22 21:27:40 2024 -0800 [Error recovery] Final eval + plot changes (#269) * partial progress on plots * progress on plots (still more to go) * small tweaks after putting plots in report * put step plots on same fig * make small changes to plots * messy own vs other plotting * change to have models next to each other, and refactor cli * add option to have reasoning in user message * add option to have answer prompt be user or system * parse mark_as_own_reasoning better * fix up run_experiments.sh * clean up own_reasoning vs other_reasoning * add gpt-4-base solver that continues assistant message * Remove unused function --------- Co-authored-by: Chan Jun Shern <[email protected]> commit 82fb4401b2b88acda979dcc9df715777274dd426 Author: James Aung <[email protected]> Date: Thu Feb 22 17:29:54 2024 +0000 Clean CurrentState for InContext RL (#282) * refactor main loop to use a single for loop * Change logging severity level * Clean CurrentState to have fewer properties * Fix random solver commit 1d0fcf608729b5ea4830faae72ca2f2698ab952f Author: James Aung <[email protected]> Date: Thu Feb 22 15:55:26 2024 +0000 Refactor main loop of in-context RL to use a single for loop (#281) * refactor main loop to use a single for loop * Change logging severity level commit 2961e233dac2b5908a181c23d899413b0d53daeb Author: Giulio Starace <[email protected]> Date: Thu Feb 22 14:55:28 2024 +0100 Coherence - missing solvers (#287) * missing gpt-4-base solver * CoT solvers for ast commit c65dac28ce91403ff4810623c00ea7540ad97927 Author: Chan Jun Shern <[email protected]> Date: Thu Feb 22 08:25:46 2024 +0000 Rolling back postprocessors :( (#286) * rollback postproc * Make private interaction mismatch error msg more informative commit 819657351194b01048fd50ee14e20415015cdd6c Author: James Aung <[email protected]> Date: Thu Feb 22 07:55:50 2024 +0000 Add anti-cot solver to In-Context RL (#279) commit 44dd342c3ab5bea59be36cec41e21cc818281f04 Author: James Aung <[email protected]> Date: Wed Feb 21 14:40:44 2024 +0000 Correct token counts for SkillAcq README (#277) commit d9a8da7a9042adef3336c47b1720021f6689bf78 Author: James Aung <[email protected]> Date: Wed Feb 21 14:40:29 2024 +0000 Correct token counts for IV README (#277) commit 3b741a791ec2724eca7405d539ab3c5fb9e006d8 Merge: 7dcd3a64 23ec5f9d Author: James Aung <[email protected]> Date: Wed Feb 21 11:03:33 2024 +0000 Merge pull request #248 from openai/james/icrl First working version of InContext RL commit 7dcd3a6493f0e4f81a71974dde6f5d3487dfc868 Author: Oliver Jaffe <[email protected]> Date: Wed Feb 21 08:03:35 2024 +0000 OpenAISolver generate from prefix (#264) * allow OpenAISolver to continue generating from prefix * return raw_completion_result in OpenAISolver SolverResult * rename start -> prefix. Check msgs length to avoid IndexError * correct typing * simplify prefix, store in fixed_start * add spaces around prefix commit 9c0804cfb9d55b5e47e644df0b0b50951e0148da Author: Giulio Starace <[email protected]> Date: Wed Feb 21 08:47:21 2024 +0100 Replace `find-letter` with `first-letter` (#274) * use first letter rather than find-letter * first letters (plural) commit bd6f1850582e2c312ae546d3da0082384f4f34c5 Author: Chan Jun Shern <[email protected]> Date: Wed Feb 21 07:45:20 2024 +0000 Bugfix: Postproc for empty/short strings (#275) * Apply handling for edge cases where string is empty or shorter than required * Add tests * Simplify RemoveQuotes using string.strip args Co-authored-by: Giulio Starace <[email protected]> * Simplify RemovePeriod using string.strip args Co-authored-by: Giulio Starace <[email protected]> * Revert RemoveQuotes to only remove matching pairs; extend tests --------- Co-authored-by: Giulio Starace <[email protected]> commit 0d05bdb3472a982050049d3e2361308a65ae3a93 Author: Giulio Starace <[email protected]> Date: Wed Feb 21 04:57:33 2024 +0100 HumanCLI solver convenience wrapper for Coherence evals (#262) * include task name in current state * allow for custom input prompt (default behaviour is unchanged) * human cli solver for track the stat * humancli wrapper for already said that * input_prompt is always a string, we provide a default * no need for `is None` check now Co-authored-by: Chan Jun Shern <[email protected]> * autoformat with black * my solvers no longer need to override input prompt * need to provide at least one SolverSpec arg for it to be recognised as one --------- Co-authored-by: Chan Jun Shern <[email protected]> commit 2f29882a14d61cca682773e0ac5adeef805c30ff Author: Giulio Starace <[email protected]> Date: Wed Feb 21 04:28:04 2024 +0100 Address evals hanging on certain samples (#260) * dont do nested multithreading * more aggressive timeout * switch back to 40 seconds commit 53085bf9c6ce76489d824556d982d885c1a78d93 Author: Chan Jun Shern <[email protected]> Date: Wed Feb 21 03:10:37 2024 +0000 add guards for solvers that subclass Solver in diff ways (#270) commit 23ec5f9dc1883088a3a5153b7b8660cfd46119f1 Author: james-aung <[email protected]> Date: Tue Feb 20 15:16:30 2024 +0000 create variant which only runs on built in gymnasium environments commit 78975fb5428855076fd3df19b09381ebe09fe6c1 Author: james-aung <[email protected]> Date: Tue Feb 20 15:11:13 2024 +0000 Add gymnasium as a dependency commit fcab6456b5b6ca7aa91083d016bebb0e6d7c77ed Author: james-aung <[email protected]> Date: Tue Feb 20 15:08:38 2024 +0000 Add Q-table initialization in QlearningSolver commit f263ba676c147d7d7aaa73e81245849c1cbaae2d Author: james-aung <[email protected]> Date: Tue Feb 20 14:54:55 2024 +0000 remove action and observation space counts from sample commit e850311a692e725035331f5511d88ac596c9f35c Author: Giulio Starace <[email protected]> Date: Tue Feb 20 15:17:01 2024 +0100 clarify task even more (#273) commit c0416c152a93992e2f318b5b38a4fc56334dbb79 Author: Oliver Jaffe <[email protected]> Date: Tue Feb 20 12:12:28 2024 +0000 Mode Collapse - Main Experiment (#265) * skeleton eval * add solver prediction evaluation * add measuring probability of predicting moves * remove probability calculation * add eval yaml, add n_samples param * add dataset creation script * add gpt-3.5-turbo dataset * add other model datasets * fix failing test, update eval call * add documentation for variant rules * improve task desc, define rules of all pieces * simplify message construction Co-authored-by: Giulio Starace <[email protected]> * rename prop -> proportion * strip now handled by postprocessors * replace dataset creation notebook with script * update datasets * remove unused SolverEval arg * pass jsonl_dir rather than relying on global args * get_model_predictions returns rather than dumping * improve documentation, make clear which rules of chess apply * default args as False * fix typo * changed large list to set * rename n_samples n_special_moves * fix previous move filtering logic --------- Co-authored-by: Giulio Starace <[email protected]> commit 2814496a01da52c1c365741134d58e15fc2c8b24 Author: Chan Jun Shern <[email protected]> Date: Tue Feb 20 09:20:37 2024 +0000 Update gpt-3.5-turbo-0125 to just gpt-3.5-turbo (#271) commit 45aeb778dc5ec9637d8160e971bfe182b4a16fcb Author: Giulio Starace <[email protected]> Date: Mon Feb 19 17:03:34 2024 +0100 Coherence - Task description adjustments (#268) * remove notion of distractors from track_the_stat * take a word from the distractor question to use in the example transcript commit 27bd87811d0b46e07f9822fbd66dd14b0c5d31a0 Author: Giulio Starace <[email protected]> Date: Mon Feb 19 15:04:42 2024 +0100 Already said that - mvp (#259) * implement build message * implement build_base_task_message * implement build_distractor_question_message * implement parse_solver_output * minor fixes * fix old implementations; distractor also evaluated now * mark violations as mistakes * no longer need rng * fill out yaml * actually use slef.task_description 🤦 * clarify instructions * adjust weighting * fix parse solver output * clearer docstring * modularize a bit * track fp and fn rate * add adversarial flag * make it extra clear when a sample is for the main task * replace rectangles and next-val-series with find-letter and which-is-heavier * balance how often we show new vs old words * .strip() handled by postprocessor; .lower() handled if match * switch to dict * use oneliner commit 2e5c35437484629327d8bd037c1fceecd2fa52ab Author: James Aung <[email protected]> Date: Mon Feb 19 07:49:42 2024 +0000 Update evals/registry/solvers/incontext_rl.yaml Co-authored-by: Chan Jun Shern <[email protected]> commit 7d71c63b5b44673488514e6ae88de81d3b70e21a Author: Chan Jun Shern <[email protected]> Date: Mon Feb 19 07:15:38 2024 +0000 Add postprocessors for Solver outputs (#245) * basic postprocessors working * Add postprocessors to subclass constructors * add to defaults * Add test for combination of postprocs * Add docstrings * Change import from just classname to full_path:classname * Log postprocessor events * Add README * Fix missing recorder in tests commit 5259ff8bb65c1668cdb8a7fba80d1e157a73e0e7 Author: james-aung <[email protected]> Date: Fri Feb 16 16:59:09 2024 +0000 cleanup commit edffa345885775aa792dc1d5b707e7d941c550cb Author: james-aung <[email protected]> Date: Fri Feb 16 16:31:27 2024 +0000 update qlearning baseline workbook commit e4a77cda0db30b8d351082e1ed8702207848295a Author: james-aung <[email protected]> Date: Fri Feb 16 15:51:52 2024 +0000 attempt at qlearning solver commit 26156168a2fcc64d9058ffe18c36edd8f67cbf05 Author: Giulio Starace <[email protected]> Date: Fri Feb 16 15:33:36 2024 +0100 Already said that (coherence) - Dataset (#254) * move data generation to scripts/ subdir * implement bulk of high-level of data generation * only missing distractor corpora * the distractors dont live in the dataset they already exist as their own separate datasets * move distractor stuff to its own module * num words as an arg * fix hanging * script and words dataset * make it executable * distractorsample dataclass * skeleton/scaffolding * missing samples_jsonl from yaml * wrong import * leverage evals.registry and evals.data * make it more robust * implement rectangles * implement next-val-series * implement ambig sentences * implement reverse word sort * make ideal just a string, not a set of strings * show distractor-specific example * add note about ifmain * remove TODO * assume running from repo root * add distractors tests commit ecd011dff7d58cbd235fa7cd8c56707703fbbf5d Author: james-aung <[email protected]> Date: Fri Feb 16 14:17:20 2024 +0000 implement random solver commit 58ff0b1bf49d1cea9b4c4c1db2101a3493fdee08 Author: james-aung <[email protected]> Date: Fri Feb 16 13:40:28 2024 +0000 update yaml commit 68377f4f259bfd1049d8ec78f1cc2b771b36ecd8 Author: james-aung <[email protected]> Date: Fri Feb 16 13:39:57 2024 +0000 keep CurrentState up to date commit efd57bd44306c6605060bb14e98400b5e238235e Author: james-aung <[email protected]> Date: Fri Feb 16 13:17:46 2024 +0000 update samples commit 5d7fd725ada83c563a66f7db7d588e83293adb51 Author: james-aung <[email protected]> Date: Fri Feb 16 13:17:31 2024 +0000 major refacto commit e20bb6e6bb4f7d69a34e8b1e7e38b70f5bc4e06a Author: james-aung <[email protected]> Date: Fri Feb 16 13:17:25 2024 +0000 add more linebreaks to prompts commit a10afc384b3a6a844481a8d6ac69e2502a6c4c9a Merge: 6ef1e812 96a6bf34 Author: james-aung <[email protected]> Date: Fri Feb 16 11:41:38 2024 +0000 Merge branch 'main' of github.com:openai/dangerous-capability-evaluations into james/icrl merge changes from main commit 96a6bf3401baa78fdd676bad9dc1ee9648cf123d Merge: e807f2dc d6e3e915 Author: James Aung <[email protected]> Date: Fri Feb 16 11:41:32 2024 +0000 Merge pull request #255 from openai/feature/20Questions Small improvements to human CLI usage commit 6ef1e8122f7d190da3f6520c2d7ee70238ac4820 Merge: 12b6b191 e807f2dc Author: james-aung <[email protected]> Date: Fri Feb 16 11:40:19 2024 +0000 Merge branch 'main' of github.com:openai/dangerous-capability-evaluations into james/icrl merge main commit e807f2dc4e85d3c8ea9c0207028ccb03caa06e25 Author: Giulio Starace <[email protected]> Date: Fri Feb 16 12:05:34 2024 +0100 simplify median state (#257) commit 12b6b191985ec82512c2f0293a411124b266148e Author: james-aung <[email protected]> Date: Fri Feb 16 10:59:53 2024 +0000 Refactor environment setup and task description generation commit 96442b5bc254f2d3e4f9ae463d99f0de601511cb Author: james-aung <[email protected]> Date: Fri Feb 16 10:43:55 2024 +0000 add baselines notebook commit 471fc9be934eb12da52e76a358cfb0603d801027 Author: Oliver Jaffe <[email protected]> Date: Fri Feb 16 09:32:19 2024 +0000 Add gpt-3.5-turbo-instruct, remove trailing whitespace from completion prompts (#256) * support gpt-3.5-turbo-instruct completion model * remove trailing whitespace from end of prompt * correct spelling of completion model * use chat_to_prefixes for rendering completion text, but strip whitespace * add documentation about removing trailing whitespace commit d6e3e915fcb9cb341538737f01d98103c30b050c Author: Andrei Alexandru <[email protected]> Date: Fri Feb 16 09:47:18 2024 +0200 Small improvements to human CLI usage commit c8dc96a9ddf13a11e5fa890896032ee446c8f868 Author: Andrei Alexandru <[email protected]> Date: Fri Feb 16 09:00:31 2024 +0200 20 questions: add shortlist variant (#253) * Add shortlist variant * Add a shortlist variant for the 'full' spec in the YAML file * Address feedback for shortlist variant * Replace how we extract the guess with regular expression * Address feedback for regex commit ca4773f0e624f66a0a8c15202b4b2328aea7d3a5 Author: Chan Jun Shern <[email protected]> Date: Thu Feb 15 17:11:09 2024 +0000 Suppress excessive 'HTTP/1.1 200 OK' logs from openai library (#252) commit b9014652eec22e2ef2e5d4b540b10c175d3c3d31 Author: Giulio Starace <[email protected]> Date: Thu Feb 15 14:13:54 2024 +0100 Coherence - Already said that distractors eval - Skeleton (#251) * scaffolding * further scaffoldign * [wip] more skeleton * cleaner skeleton * more skeleton, add utils * fix name * implement agg metrics * set constant for num turns * add docstrings * add mocks * fix flow to enable tracking perf on distractor task * rename violation to violation_occurred * move convo loop to its own func * make elif more readable * remove redudnant empty messages * address constants commit b4875c129c61b35aea0d1cf7c3d1cac262c4654d Author: Andrei Alexandru <[email protected]> Date: Thu Feb 15 10:11:25 2024 +0200 20 questions: adding features I (#250) * Add features: logit bias, 'skip' option, handling incorrect guesses and rule violations * Add counter for gamemaster 'skip' replies * Remove finished TODOs commit b97ec783ec75483148f63c5e5c109fba7e9430c9 Author: james-aung <[email protected]> Date: Wed Feb 14 15:37:35 2024 +0000 calculate rolling reward commit 8797064be11d6fda3ab37dc529e81fb409321729 Author: james-aung <[email protected]> Date: Wed Feb 14 14:12:12 2024 +0000 track which steps episodes end commit 1b7661bea9c41902b5575a4ce05c8a13b338b535 Author: james-aung <[email protected]> Date: Wed Feb 14 14:04:44 2024 +0000 reintroduce backup max_steps commit 2466a1df070f7bbec90c26624692344bd6ea699c Author: james-aung <[email protected]> Date: Wed Feb 14 13:58:14 2024 +0000 sample now runs until context limit is reached commit 557ddddb33914a6e5e778d684b9a76f5a014c8ea Author: james-aung <[email protected]> Date: Wed Feb 14 12:50:38 2024 +0000 remove n_trials concept commit 5ee3fff6f1f04b535058da26de342401af1c6726 Author: james-aung <[email protected]> Date: Wed Feb 14 12:05:53 2024 +0000 adjust task description based on feedback commit a236da4c8bd8924343b1818d261951349c02b365 Author: james-aung <[email protected]> Date: Tue Feb 13 17:30:12 2024 +0000 tidy up commit 9a5a6f27a49e608df693c0631c0788d6744e2966 Author: James Aung <[email protected]> Date: Tue Feb 13 16:59:42 2024 +0000 Update evals/elsuite/incontext_rl/requirements.txt commit 2f2609043b5293ffdc6fe8e47be33c2c71b4ae97 Author: Oliver Jaffe <[email protected]> Date: Tue Feb 13 12:34:08 2024 +0000 Mode Collapse - Chess Framework (#247) * chess game skeleton logic * implemented algebraic notation parser * logic for moving pieces * prevent moves that threaten the king * add support for castling moves * add support for en passant moves * add support for pawn double moves * fix pawn moves, don't allow them to capture forwards * add support for promotion moves * add method to find piece on board * compute normal moves before promotions, otherwise no promotions are possible * add testing for our framework vs. python-chess * verbose option for test * add missing typing * skip board testing to stop CI failing * prevent pawns promoting to kings * simplify notation parsing. Assume full start and end position is given * rename notation function to intended name * rename board function to more descriptive name * rename possible_moves -> possible_transformations, since wasn't using Move object * move import within test to prevent CI failing * Piece object now calculates its possible moves * add file containing move variants * rename pieces -> piece_id_to_instance * add running test upon executing file. rename testing vars * notation parser no longer requires player_id * rename notation parser functions to reflect that move object is being processed * removed unused initial_board and _find_pieces func * add documentation for _update_board * remove unnecessary validation of move * split some get_piece_moves logic into separate function * rename functions to indicate if transformation or move is returned * refactor to avoid circular imports. now Board isn't passed to notation parser or Piece * add documentation to board classes * nit: update typing commit 36c7faac0ab15e53e523f14d801fef1306f7ce99 Author: Giulio Starace <[email protected]> Date: Tue Feb 13 11:27:25 2024 +0100 Coherence - Explicit State baseline solver (#249) * implement eval side * fix serialization issue * make it a dict explicitly * implement explicitstate solver * idk how this crept back in * rename to track the stat * fix imports and ids * move helper functions to utils.py * keep logger, not logging commit b10f23c436b64562068c3c880dbd3494ece7da95 Author: Giulio Starace <[email protected]> Date: Tue Feb 13 10:53:35 2024 +0100 Coherence - Implicit State Tracking (#246) * implement turns, add task desc * append messages * implement task_fn and parse_solver_output * singleturn was missing Message construction * fill in prompts * remove fulfilled todos * 500 samples * avoid info logs from httpx * integrate singleturn task desc * use stateless solver to avoid state leak in singleturn * make a submodule for the prompts * further modularize prompts * remove extra space in warning * fix lsit rendering in singleturn example * more metrics * switch to aritmean median * strip notion of singleturn/multiturn dichotomy * fix serialization issue * rename this to implicit state tracking * get existing logger and use that instead of logging * add note to todo * rename to track_the_stat * finish renaming commit 066a6bb011e50041e93b72ecf82cb9d6446e691a Author: Andrei Alexandru <[email protected]> Date: Mon Feb 12 10:47:34 2024 +0200 Initial PR for 20 questions (#240) * Twenty questions MVP * Refactor to conversation loop * Fix tests to use Message class * Some questions have clarifications after the ?, like (eg.) * Log word complexity metric * Tweak guesser prompt * Add lexicon from Maddela and Xu, 2018 * Add dev5, remove completed TODOs * Add score hint in guesser prompt * Add flexible max score to prompt * Consistent naming for max_questions * Move from accuracy to score * Add scoring, TODOs, fix dev5 num_samples * Remove test file, it's no longer needed * Shuffle before we sample * Move from specific gpt-4 version to -turbo alias * Add fall-back condition commit f525f70c3f55533c9a328feef3fa69e59c05a03f Author: Chan Jun Shern <[email protected]> Date: Mon Feb 12 08:41:13 2024 +0000 Add "System: " prefix when rendering system messages for base model (#244) * Modify base model msg rendering; OAISolver uses 'System:' prefix * Fix failing tests commit d99e6c46e512ab306b800050692a041f4179eb0c Author: james-aung <[email protected]> Date: Sun Feb 11 20:19:35 2024 +0000 change to 50 steps commit c71cb3a6b096a15612cdc54b6e5254c73d1fcdd0 Author: james-aung <[email protected]> Date: Sun Feb 11 19:22:26 2024 +0000 adjust variants commit c82ee84b33f4ac821275d00c169674075eca4642 Author: james-aung <[email protected]> Date: Sun Feb 11 19:21:49 2024 +0000 Cast cumulative_reward to float in InContextRl eval commit 9dd1ea31720e973f73ff9f3ae1498b835cda26a7 Author: james-aung <[email protected]> Date: Sun Feb 11 18:36:44 2024 +0000 fixed dataset by removing duplicate sample commit e17e8e87c4d93e8268d526708dee6a137a04720e Author: james-aung <[email protected]> Date: Sun Feb 11 18:19:03 2024 +0000 split out a explanations variant commit a0e2278fc1988b9eae4d483a9b4bf1b5fbe90bf9 Author: james-aung <[email protected]> Date: Sun Feb 11 18:18:37 2024 +0000 add cliffwalking and 10armed bandit envs commit d5725e76a69cd141e684619212dda0b8444d3193 Author: james-aung <[email protected]> Date: Sun Feb 11 18:18:10 2024 +0000 pass through explanined observation to step messages commit 20268cbc85977238aeeda3a24a87a80af4a54fd9 Author: james-aung <[email protected]> Date: Sun Feb 11 18:17:43 2024 +0000 update prompts commit 8a399ea54121b6f592aae5344a7d1e6347c09992 Author: james-aung <[email protected]> Date: Sat Feb 10 19:33:45 2024 +0000 Update to 60 steps commit f99a6130366fda51dcb45f3e1703666e1a688dc6 Author: james-aung <[email protected]> Date: Sat Feb 10 19:33:32 2024 +0000 Suppress annoying logging commit 13c05dc5b4956e10d26c3947510974d39ed1aaf5 Author: james-aung <[email protected]> Date: Sat Feb 10 19:11:12 2024 +0000 cleanup commit ea7d93f875e154c64ed0b12f800eb7a1bba6cea5 Author: james-aung <[email protected]> Date: Sat Feb 10 15:12:36 2024 +0000 add suport for multiple trials per env commit 8dd4ebe98b0795a611f95a6501d799fa6c414bd8 Author: james-aung <[email protected]> Date: Sat Feb 10 15:12:18 2024 +0000 toggle explanations on commit 300902e1233318deda0b467bb979ab2d98fa4ce3 Author: james-aung <[email protected]> Date: Sat Feb 10 15:12:06 2024 +0000 update samples commit e1a5499394394fb6a62ab4c4759f5ee024598adb Author: james-aung <[email protected]> Date: Sat Feb 10 14:05:10 2024 +0000 Update incontext_rl.yaml with new n_steps value commit 1a773de8876be16e5f9715f39adb1eaa1563b5c1 Author: james-aung <[email protected]> Date: Sat Feb 10 14:05:04 2024 +0000 add exlanations to dataset commit d9dd017a75eec5e387624ae00323d10130393325 Author: james-aung <[email protected]> Date: Sat Feb 10 14:04:57 2024 +0000 allow for explanations commit bfd80b7f864d8388e17798c0822c3b68867020f1 Author: james-aung <[email protected]> Date: Sat Feb 10 13:57:08 2024 +0000 ffix spacing commit 7188886ec34ceb57f84e362400019786554eb775 Author: james-aung <[email protected]> Date: Sat Feb 10 13:47:52 2024 +0000 add action parsing commit 678957f66d2253d0740c2bffa111a66a19d45867 Author: james-aung <[email protected]> Date: Sat Feb 10 13:47:42 2024 +0000 remove old notebook commit 21132e552a46466efd5f97ca6aac5a3c5cbccf2a Author…
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hey everyone,
For some reason, when running on linux distro (wsl2 or ubuntu) the
pip install -e .
fails while it works on other operational systems. I've been getting an error saying its built backend is missing on the pyproject.toml.I'm not sure if anyone else is experiencing this or if it's just me. I have, however, just tried on 2 devices and had the same error so I decided to open an issue.
here is the error message:
The text was updated successfully, but these errors were encountered: