Non atomic rewrite of config

[lukemarsden]

I sometimes see errors in the logs about malformed yaml. This must be because we're not atomically rewriting the config.

cc @jasonrichardsmith