-
Notifications
You must be signed in to change notification settings - Fork 213
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Exploring a redesign and discussing how a modern Rust configuration library should behave #111
Comments
Just a comment describing how I tend to setup configuration:
My pain points:
|
I love how struct Settings {
foo: String,
bar: Option<BarSettings>,
bar: Option<BarSettings>,
}
struct BarSettings {...}
struct BazSettings {...} And then my config looks like this: foo = "hello world"
[bar]
item1 = 1234
item2 = "asdf" If I create If you try to merge in something like I see a couple potential ways to address this:
I suppose a similar situation could occur when getting config values from the command line. I think the larger questions is what does it mean to get config values from a non-hierarchical source, and then later merge them with a hierarchical source? |
The idea is that you would override using the key
It seems we need to think hard on how CLI arguments tie into this. I have a fairly good idea on how to tie clap in normally but I hadn't thought of how to tie config into clap as a "default values provider". |
@mehcode Sorry, edited parent comment. If I create a config from the above config file, I should get I understand the separator trick, but I AFAIK it's not possible to use that to un-define a table (in this case, |
@mwillsey Hmm.. Un-define in general is a hard concept. We could probably allow a JSON string to be merged in for a table or array. Viper (go config lib) does that.. but that only can get you an empty map. Not a Un-setting from a string is probably a lot easier as you could just merge in a string and tell config to treat it as JSON or TOML or something. |
Considering merging, I would like to see "action" keys, which will make use of special characters. These keys should allow altering config when merged. For example, "remove$app.bar" to remove value completely. Or "add$app.bar"=[1, 2, 3] to add values into array. |
I use Therefore I guess the interface doesn't matter to me that much, though I'd really like having more informative error messages and being a bit more robust and well defined about how the merging and such works. |
I also use this crate for its ability to merge from multiple sources, including defaults, and to deserialize to a struct. I guess if I were to do, I'd do the same, with some kind of format provider which would feed with Option. My only problem is the case (#65). I heavily rely on sub-objects for namespacing, because IMHO some (theoretically all) modules should not know about others, and as such I'm not fan of 12 factor's reliance on a flat list of key-value pairs. Besides with docker configs and secrets, it's easy to provide different config files for different envs. Anyway, I'd say either a project relies on env vars only, or on config files only, but not both (I gladly receives counter-examples). Serde allows for renaming struct fields, so for env vars |
I could imagine a project where one would have most of the configuration inside a config file, but set passwords through environment variables. I think we have some such uses in the company, though not in Rust. As for the case, I actually find the fact the config is case insensitive as an advantage. Not a huge one, but it feels more convenient from the user perspective. |
Thanks for the use case. I think you imply that both the config from the file and the passwords end up in the same list. As I prefer to divide config by modules, I don't see how both needs could fit together. If I had to accept env vars, I guess I'd have a dedicated struct for available env keys, and then I'd put these values into the config structs. Ultimately this dedicated struct would be similar to what structopt does, i.e. env var and cli arguments serve the same goal. |
Yes, I like to have one „big“ config structure and then I hand out parts of it to submodules of the application. I tend to consider configuration reload to be transactional, so I don't want to load part of new configuration and then fail on another sub-config. Nevertheless, even if I didn't have that, let's say I have something like „update source“ ‒ a place where updates for whatever data the application holds. #[derive(Deserialize)]
struct Updates {
base_url: Url,
username: String,
password: String,
} This is definitely supposed to come to the same place in the application, so it should be the same config. But the url will be in config file, probably distributed together in a docker image or deb package or whatever. The password will be overwritten by an environment variable. |
Sorry for jump in between, i never tried config-rs, just wanted to share quite famous way to handle config. https://github.com/spf13/viper Let’s you use config without worry of from where value is coming from, it can be from file/env/key-value-store etc. i love that application doesn’t need to care about how value being set. |
One use case for me would be able to use
I'd personally be happy passing a slice of |
Regarding the first example, I'm not a fan of manually In fact, I'd like to see an option for the separator to get even smarter: I'd like to have an environment variable like pub struct Settings {
foo_bar: SubSettings
}
pub struct SubSettings {
baz_blah: i32
} I know this is a long-shot because we have to map to a key string, which has no knowledge of its eventual location in a struct. |
The way I see it, I am looking for two things from the config file :
I think there are two ways of addressing point 2:
I'd like to know what other think:
|
Let me refer to your concern with point number 1. You do not need to patch values from environment variables directly. What you can do it to keep various versions of config files in VCS named for example
What you'd keep in env variables is just the name of environment, here : Ideally you'd want to write in settings files "down the waterfall" only those lines you'd want to overwrite (here : only What I propose is how we solve such issues at my company (and in .NET Core world from which I come from) so you could take a look at IConfiguration inteface (and its implementation) in .NET Core which is heavily battle-tested and proven to work in many circumstances. |
And needing to update all files whenever you need to change a config. Which can be easy to forget. |
Just to throw my bone in the ring I've been fighting against the state of config in Rust this last week and came up with this as my personal ideal workflow:
The methods defined by the derive would include the following:
And more, obviously. What I really like about this is the derive and builder pattern lets you incrementally add sources and eventually have this opaque configuration struct that can do a lot behind the scenes - I can see constructors that can take event loop handles to use async or threadpools for the IO and these builder methods can be generally complex because I can imagine a lot can be done with constexpr at compile time to figure out what the final struct is supposed to actually look like. Ultimately you would only be doing real work in the build() function using a bunch of constexpr prefill. But for a user all you are doing is making a plain old data struct, using a derive macro, and building your config with the options you want. No other boilerplate. To write to it you just set properties on the struct and call |
Hello Your builder approach looks kind of nice on the high level. But there are some things that don't add up for me: If you already want your struct to be If you only care about these new methods, wouldn't an extension trait (something like itertools or similar does to add more methods to all the iterators) on top of // In the lib:
pub struct Config: DeserializeOwned {
fn new() -> ConfigBuilder<Self> { ... }
}
// The user:
#[derive(Deserialize)]
struct MyConfig {
params: Vec<String>,
}
impl Config for MyConfig {} // Yes, empty here
I'm not sure if integration with threadpools should be in scope of this library. I mean, if there's a library that can read bunch of formats, combine different sources together and spits out parsed data structure, that's great. Should watching files and realoading be done in the same library, or in some that builds on top? I don't really want to pull in the whole tokio to read a 20 lines long config file 😈.
That sounds a bit scary. First, I believe most applications don't modify its own configuration. At least servers don't. And it seems to be not obvious where it would store the changes if, for example, the whole config was read from the environment variables. And if save takes a path to the file, does it provide anything on top eg. serde-json, if you structure already implements I don't see what exactly you want to do in So, in general, the composition by the builder pattern seems nice (nicer than the I also wonder what scope the library should have. For me it would make sense to split the parsing/merging/decoding of configuration from other features, like file change watching (both because one might not to want it, and because there's a million ways how you may want to trigger your config reload, or where it happens and how the changes are propagated back into the program; as the author of spirit, I know how much complexity there is :-| ). |
Why? The way I imagine watching working, is when a fule changes, it just reloads the full config.
Save on close doesn't necessarily require save on drop. And having the save method separate givea the developer more control, and the ability to handle errors. |
I see, so the purpose of the derive is to store additional state in the config state? Couldn't the same thing be accomplished with a wrapper
Exactly. If that is desired behavior, the user of the library can just implement |
First, AFAIK, derive macros don't allow you to inject more fields. They allow you to add code, but outside of the struct (eg. make Second, why would I ever want, as the user of the library, to intermix the config structure (the data loaded from the config file) with the manager? Shouldn't the manager be some kind of object that does the tracking and provides me/contains/derefs to the actual config I've decided to use? Putting both into the same thing conflates two things into one. I can certainly want to pass a config structure to a function, but not pass the manager into the same function. |
A little off-topic, but how about a macro that embeds the configuration provided at compile-time into your code? Essentially, during compile time (perhaps in CI), the configuration can be provided through the same merge-semantics (and combining multiple configurations) and provided as a constant struct, making value accesses hard-coded? Idk just a random idea. Maybe that combined with env vars during build time would be useful in a few use-cases (especially in embedded scenarios), considering runtime performance would be improved significantly, since the compiler can now optimize for constant values. |
+1 for this one. I found re-creating large parts of config.rs manually to achieve clear precedence rules flags > env > toml. |
Adding to this issue, but I might also open a separate one. I've just found out about this crate, and although it looks like it covers part of what I need, I'm actually put off by the sheer number of mandatory dependencies. The support for the various serialization formats are behind Cargo features as one would expect, but it seems (or did I misunderstood something?) that the various sources are all mandatory. In particular, web/online ones like HTTP via |
I moved this issue to the project so we can consider it as well (see here for more details). If you, as a commenter in this issue, want to contribute to the effort of re-thinking config-rs, please open issues for your ideas (see the "Help wanted" section here). |
Two years later! - How're the plans progressing? |
Hi, just wanted to add that allowing Humanized numbers would be great, maybe leveraging the parse_size crates parse_size function to convert 1KB into 1024 etc |
This library has gotten quite stale over the last year and I've been loathe to come back to it mostly because I don't like it. It's complex to use, complex to develop, etc. I apologize for letting it atrophy.
I'd like to thank you everyone who has contributed or submitted a PR ( especially those who've done so in the last several months and have just sat there ). It's not that I don't want to merge in your work, it's that if I come back to this I'll want to do too much and I haven't had the time available to do that.
With that out of the way I want to explore how a configuration library could behave in Rust, optimally, using some example use cases as a jumping off point:
Thoughts? How would you want
config
to work?Are there any pain points in the current design you'd like to discuss?
The text was updated successfully, but these errors were encountered: