[WandB] Push axolotl config to top level wandb files

[hamelsmu]

@teknium1 has told me that one annoying thing is finding the axolotl configs in wandb. This really drives me insane too. This PR pushes the config to the top level files under each experiment run in wandb (and also keeps it in artifacts).

We copy the axolotl config into a new file called axolotl_config_{tempfile_uid}.yml so we do not collide with the existing config.yml that wandb itself pushes to the files.

This is how it looks now (example is from this run)

[NanoCode012]

May I ask if this will now save the artifacts in two locations (prior location and top-level) or only top level? If the former, should we adjust to only the latter?

[hamelsmu]

May I ask if this will now save the artifacts in two locations (prior location and top-level) or only top level? If the former, should we adjust to only the latter?

Yes artifacts have some fancy features like lineage, versioning, etc. It could be that people want to use that. The only issue with artifacts is that its very buried in the application. I think that most people running axolotl do not care about any kind of lineage of a config file. Lineage is something that makes sense for data or for models.

So @NanoCode012 I removed the artifact version, good suggestion.

[tcapelle]

This is a good idea, it would be good to save the "modified" config in case the user does some computation (or the library). I am also thinking in the case one would inject changed parameters from the W&B UI.

[hamelsmu]

@tcapelle can you elaborate perhaps with an example I'm not sure I follow

[tcapelle]

Pseudo Code

# load the config from yaml
parsed_config = load_cfg(default_file)

wandb.init(...., config=parsed_config)

# overwrite params with the config from a sweep or launch
config = wandb.config

# train with axolotl
train(config)

# inject from wandb
save_yaml(config)

[hamelsmu]

I don't think it's a good idea to inject wandb config into the axolotl one

I don't like the vice versa either too much danger of collisions

Wandb already auto logs a config because axolotl is using HF Trainer under the hood which is logged as config.yml under files.

I think to make the case we would need axolotl to be a first class way to run jobs in wandb. Axolotl users will want to use something closer to the native axolotl interface. Happy to talk to you about that 😃 could make it happen in launch or something else