Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Quality of life improvements for experiments #9

Open
5 of 6 tasks
ejnnr opened this issue Sep 13, 2023 · 0 comments
Open
5 of 6 tasks

Quality of life improvements for experiments #9

ejnnr opened this issue Sep 13, 2023 · 0 comments

Comments

@ejnnr
Copy link
Owner

ejnnr commented Sep 13, 2023

This issue is for tracking various features that could make running large-scale experiments more convenient. Unclear which ones are actually worth implementing, will probably add them over time whenever one seems like it would actually be helpful right now.

  • Allow loading a configuration from a yaml file.
  • Launcher for sweeps, ideally with Slurm integration. I was using the Hydra submitit plugin for this in an earlier version of this code base but then switched away from hydra. It clearly doesn't make sense to reimplement all of hydra's complexities, but we could for example just have a way of sweeping over experiment configs (previous point).
  • When training a detector, the type of anomaly (like what the backdoor trigger is) usually doesn't matter but you still need to specify it for no good reason.
  • Have an entrypoint function and register it with pip, so you can run cupbearer ... instead of python -m cupbearer.scripts...
  • Checkpointing
  • Make sure we don't accidentally overwrite files from previous run (e.g. check whether the output directory already exists)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant