Consider env vars when unmarshalling #188

aeneasr · 2016-05-17T16:59:47Z

Is this possible?

UPDATE: yes it is, see #188 (comment)

type Config struct {
    BindPort int `mapstructure:"port" yaml:"port,omitempty"`
}

var c Config

// ...
viper.AutomaticEnv()

if err := viper.ReadInConfig(); err != nil {
    // ...
}

if err := viper.Unmarshal(&c); err != nil {
    // ...
}

// I want c.BindPort == viper.Get("PORT") == os.Getenv("PORT")

rybit · 2016-10-22T00:09:42Z

I am having the same problem, it isn't fixed directly by the #195. You have to register defaults it seems before viper will load the values. I have been experimenting with some terrible reflection code to go set defaults from a given structure, but I'd like to keep reflection out where possible.

Consider the following code:

type Config struct {
  CAFiles  []string `mapstructure:"ca_files"
  CertFile string   `mapstructure:"cert_file"
  KeyFile  string   `mapstructure:"key_file"
  Servers  []string `mapstructure:"servers"`
}


func configuredViperInstance() *viper.Viper {
  v := viper.New()
  v.SetConfigName("config")
  v.AddConfigPath("./")

  v.SetEnvPrefix("test")
  v.SetEnvKeyReplacer(strings.NewReplacer(".", "_"))
  v.AutomaticEnv()

  if err := v.ReadInConfig(); err != nil && !os.IsNotExist(err) {
    panic(err)
  }
  return v
}

func main() {
  noDefaults := configuredViperInstance()
  config := new(Config)
  if err := noDefaults.Unmarshal(config); err != nil {
    panic(err)
  }

  fmt.Printf("%+v\n", config)
  fmt.Printf("config-cert_file: %s\n", config.CertFile)
  fmt.Printf("cert_file: %s\n", noDefaults.GetString("cert_file"))

  fmt.Println("-------------------------")

  withDefaults := configuredViperInstance()
  withDefaults.SetDefault("cert_file", "")
  config = new(Config)
  if err := withDefaults.Unmarshal(config); err != nil {
    panic(err)
  }
  fmt.Printf("%+v\n", config)
  fmt.Printf("config-cert_file: %s\n", config.CertFile)
  fmt.Printf("cert_file: %s\n", noDefaults.GetString("cert_file"))
}

I then call it passing in this json

{
  "servers": ["server-1", "server-2"],
  "key_file": "key-file"
}

TEST_CERT_FILE="env-certfile" go run nested_conf_testing.go

I get the response

&{CAFiles:[] CertFile: KeyFile:key-file Servers:[server-1 server-2]}
config-cert_file:
cert_file: env-certfile
-------------------------
&{CAFiles:[] CertFile:env-certfile KeyFile:key-file Servers:[server-1 server-2]}
config-cert_file: env-certfile
cert_file: env-certfile

aeneasr · 2016-10-22T10:01:58Z

@rybit you need to do

type Config struct {
    BindPort int `mapstructure:"PORT" yaml:"port,omitempty"`
}

var c Config

// ...
viper.AutomaticEnv()

viper.BindEnv("PORT")
viper.SetDefault("PORT", 4444)

if err := viper.ReadInConfig(); err != nil {
    // ...
}

if err := viper.Unmarshal(&c); err != nil {
    // ...
}

to get this working.

aeneasr · 2016-10-22T10:02:29Z

If BindEnv or SetDefault is missing, it won't work with the latest beggining with PR #195

benoitmasson · 2016-10-22T15:37:25Z

@rybit I don't think this is a bug, I'd say that this is the expected behavior: how should Viper know that a struct field has to be mapped to an environment variable, if you don't tell it to?

If you ask for config value foo with viper.Get(), Viper knows it has to look for a variable FOO in the environment and viper.AutomaticEnv() will be sufficient to handle that. But on the other side, viper.AllSettings() (used by viper.Unmarshal()) can't be aware that a config value for foo is expected (unless all environment values are added to the map, which wouldn't be reasonable).

As for me, automatically adding defaults or environment binding with reflection is the way to go, if done carefully. You could then use structure tags to have the programmer define his own default value, which is very handy and flexible (I do it myself :-) )

@arekkas From what I remember, if AutomaticEnv is on, then SetDefault() is enough (this tells Viper that such a configuration value exists). BindEnv is useful only when AutomaticEnv is off, and in that case, SetDefault() shouldn't be needed.
Are you sure both are necessary? [EDIT: just saw your other post in the PR, I'll investigate this]

burdiyan · 2018-01-04T15:21:08Z

I wonder if it's possible to make @arekkas' example working without explicitly doing BindEnv and just doing AutomaticEnv.

Normally we use the following workflow in our applications:

Define global configuration structure as a Go struct.
Expose flags that can modify this configuration and bind the flag set to viper.
Allow to modify the configuration via file, flags or env variables.

Inside the application code we only deal with the config struct (to avoid dealing with arbitrary string keys in Viper). And we explicitly pass the config, it is not a global structure.

Basically the default values of the config are defined in flags. So I would like to avoid explicitly doing BindEnv for each of the env vars. It seems like it should be doable to have this with AutomaticEnv directly.

burdiyan · 2018-01-04T15:24:43Z

Oh, it seems like doing viper.SetEnvKeyReplacer(strings.NewReplacer(".", "_")) makes it possible.

So maybe adding some documentation about it can make this issue fixed.

from-nibly · 2018-05-09T19:52:45Z

@burdiyan were you able to confirm that your solution with the SetEnvKeyReplacer works? I am unable to get it to work in my project. Maybe there is another thing I am missing?

calling viper.GetString("foo.bar") gets me the value but calling viper.UnmarshalKey("foo", &cfg) uses the value found in my config file.

from-nibly · 2018-05-11T17:20:30Z

nvm I figured out you can fix it with this work around

for _, key := range viper.AllKeys() {
	val := viper.Get(key)
	viper.Set(key, val)
}

aarondl · 2018-06-22T17:15:10Z

@from-nibly did you end up using BindEnv or SetDefault to ensure you had keys to reference? I'm looking at this problem and it seems like I'm going to have to crawl through the env variables and set defaults/values for them in order to make Unmarshal work correctly.

	viper.SetEnvPrefix("application")
	viper.SetEnvKeyReplacer(strings.NewReplacer(".", "_"))
	viper.AutomaticEnv()

Even with this setup the methods AllKeys and AllSettings return nothing, which Unmarshal uses directly to try to prepare a structure for the mapstructure library to work.

from-nibly · 2018-06-22T17:27:14Z

here is the full code I used to create a viper instance that I could use to Unmarshal an object that used env variables

func LoadConfig() *viper.Viper {
	conf := viper.New()

	conf.AutomaticEnv()
	conf.SetEnvPrefix("ps")
	conf.SetEnvKeyReplacer(strings.NewReplacer(".", "_"))

	conf.SetConfigName("application")
	conf.AddConfigPath("/etc/my-app")
	err := conf.ReadInConfig()

	if err != nil {
		switch err.(type) {
		default:
			panic(fmt.Errorf("Fatal error loading config file: %s \n", err))
		case viper.ConfigFileNotFoundError:
			logger.Warning("No config file found. Using defaults and environment variables")
		}
	}

	// workaround because viper does not treat env vars the same as other config
	for _, key := range conf.AllKeys() {
		val := conf.Get(key)
		conf.Set(key, val)
	}

	return conf
}

aarondl · 2018-06-22T18:42:43Z

Odd. I do the same thing on the latest master branch and I still wind up with nothing being loaded into my struct. There's nowhere you actually define defaults? Or maybe your config file has empty values in it? When I try this AllKeys is totally empty unless I put things in the config file.

from-nibly · 2018-06-22T18:47:51Z

oh yes sorry just looked at it again. I define each property as "empty string" in my config file (which gives me the list of property names in AllKeys)

krak3n · 2018-06-25T09:09:24Z

Edits: Syntax

I've also run into this problem so I wrote a quickie (and perhaps dirty) recursive function which manually calls BindEnv on a struct so you can have a partially complete config file:

func BindEnvs(iface interface{}, parts ...string) {
	ifv := reflect.ValueOf(iface)
	ift := reflect.TypeOf(iface)
	for i := 0; i < ift.NumField(); i++ {
		v := ifv.Field(i)
		t := ift.Field(i)
		tv, ok := t.Tag.Lookup("mapstructure")
		if !ok {
			continue
		}
		switch v.Kind() {
		case reflect.Struct:
			BindEnvs(v.Interface(), append(parts, tv)...)
		default:
			viper.BindEnv(strings.Join(append(parts, tv), "."))
		}
	}
}

Usage:

// Config holds configuration
type Config struct {
	Log    Log    `mapstructure:"log"`
	DB     DB     `mapstructure:"mongo"`
	Server Server `mapstructure:"server"`
}

// Log configuration
type Log struct {
	Format string `mapstructure:"format"`
}

// DB configuration
type DB struct {
	URL    string `mapstructure:"url"`
	Name  string `mapstructure:"db"`
}

// Server holds HTTP server configuration
type Server struct {
	Listen          string        `mapstructure:"listen"`
	ShutdownTimeout time.Duration `mapstructure:"shutdown_timeout"`
}

// set up env key replacer and prefix if applicable
func init() {
	viper.SetEnvKeyReplacer(strings.NewReplacer(".", "_"))
	viper.SetEnvPrefix("MY_APP")
}

// Builds config - error handling omitted fore brevity
func config() Config {
	c := Config{} // Populate with sane defaults
	viper.ReadInConfig()
	BindEnvs(c)
	viper.Unmarshal(&c)
	return c
}

// Get and use config
func main() {
    c := config()
    fmt.Println(c.DB.URL)
}

I guess you could also do this in init:

func init() {
	viper.SetEnvKeyReplacer(strings.NewReplacer(".", "_"))
	viper.SetEnvPrefix("MY_APP")
	BindEnvs(Config{})
}

natefinch · 2018-06-29T18:08:25Z

@krak3n - thanks for the hack to get this to work. I was considering doing the same thing, and happy to have someone else work out the fiddly bits.

For reference, mine defaults to use the lowercase name of the field instead of requiring the mapstructure tag (since almost all of mine are just straight strings):


func bindEnvs(v *viper.Viper, iface interface{}, parts ...string) {
	ifv := reflect.ValueOf(iface)
	ift := reflect.TypeOf(iface)
	for i := 0; i < ift.NumField(); i++ {
		fieldv := ifv.Field(i)
		t := ift.Field(i)
		name := strings.ToLower(t.Name)
		tag, ok := t.Tag.Lookup("mapstructure")
		if ok {
			name = tag
		}
		path := append(parts, name)
		switch fieldv.Kind() {
		case reflect.Struct:
			bindEnvs(v, fieldv.Interface(), path...)
		default:
			v.BindEnv(strings.Join(path, "."))
		}
	}
}

supershal · 2018-08-15T23:13:46Z

all the options suggested above either required to set config file with empty values or bind each environment variables. These options are prone to errors.

I tried following simple workaround based on comments from this issue. It does not require to bind env values or set defaults or set file with all config set to EMPTY VALUE or reflection.

type Config struct {
	Name string `yaml:"name"`
	Port int `yaml:"port"`	
}
func NewConfig() *Config {
	return &Config{Port: 8080, Name: "foo"}
}

var cfgFile = "/tmp/overwrite.yaml" // contents: name: bar
var config *Config
func initConfigE() error {
	v := viper.New()
	// set default values in viper.
	// Viper needs to know if a key exists in order to override it.
	// https://github.com/spf13/viper/issues/188
	b, err := yaml.Marshal(NewConfig())
	if err != nil {
		return err
	}
	defaultConfig := bytes.NewReader(b)
	v.SetConfigType("yaml")
	if err := v.MergeConfig(defaultConfig); err != nil {
		return err
	}
	// overwrite values from config
	v.SetConfigFile(cfgFile)
	if err := v.MergeInConfig(); err != nil {
		if _, ok := err.(viper.ConfigParseError); ok {
			return err
		}
		// dont return error if file is missing. overwrite file is optional
	}
	// tell viper to overrwire env variables
	v.AutomaticEnv()
	v.SetEnvPrefix(envVarPrefix)
	v.SetEnvKeyReplacer(strings.NewReplacer(".", "_"))

	// refresh configuration with all merged values
	config = &Config{}
	return v.Unmarshal(&config)
}

krak3n · 2018-08-16T09:32:16Z

@supershal nice approach, nice way to avoid reflection. I'll see if this can work with toml formatted configs tool 👍

unmultimedio · 2019-09-11T19:55:43Z

Hello, any updates on this? getKeys func still does not read from ENVwhen no config file or defults are set.

sagikazarmark · 2019-09-12T01:06:38Z

Please see and follow the issue above. ☝️

tunhuit · 2021-06-17T14:53:06Z

I have the same issue but with flags. Unmarshal() works only with value from config file, it doesn't check flags.

Kavit900 · 2022-06-02T22:06:24Z

Hi Viper team, so I have set environment variables in my docker container and I am trying to use viper to get those values and then use unmarshall but it doesn't seem to work, does viper only works if I have a separate environment file?

76creates · 2022-06-28T15:50:57Z

all the options suggested above either required to set config file with empty values or bind each environment variables. These options are prone to errors.

I tried following simple workaround based on comments from this issue. It does not require to bind env values or set defaults or set file with all config set to EMPTY VALUE or reflection.

type Config struct {
	Name string `yaml:"name"`
	Port int `yaml:"port"`	
}
func NewConfig() *Config {
	return &Config{Port: 8080, Name: "foo"}
}

var cfgFile = "/tmp/overwrite.yaml" // contents: name: bar
var config *Config
func initConfigE() error {
	v := viper.New()
	// set default values in viper.
	// Viper needs to know if a key exists in order to override it.
	// https://github.com/spf13/viper/issues/188
	b, err := yaml.Marshal(NewConfig())
	if err != nil {
		return err
	}
	defaultConfig := bytes.NewReader(b)
	v.SetConfigType("yaml")
	if err := v.MergeConfig(defaultConfig); err != nil {
		return err
	}
	// overwrite values from config
	v.SetConfigFile(cfgFile)
	if err := v.MergeInConfig(); err != nil {
		if _, ok := err.(viper.ConfigParseError); ok {
			return err
		}
		// dont return error if file is missing. overwrite file is optional
	}
	// tell viper to overrwire env variables
	v.AutomaticEnv()
	v.SetEnvPrefix(envVarPrefix)
	v.SetEnvKeyReplacer(strings.NewReplacer(".", "_"))

	// refresh configuration with all merged values
	config = &Config{}
	return v.Unmarshal(&config)
}

reading this I came to understanding that SetEnvKeyReplacer is doing complete opposite of what I taught it did, i was setting it to strings.NewReplacer("_", "."), might be good idea to add the explanation/example to the function

wsgomes · 2022-10-11T01:49:46Z

This is my solution:

type Configurations struct {
	Database DatabaseConfigs
}

type DatabaseConfigs struct {
	User string
	Pass string
	Host string
	Port int
	Name string
}

var configs Configurations

func init() {
	viper.SetConfigName("config")
	viper.AddConfigPath("config")
	viper.SetConfigType("yml")

	if err := viper.ReadInConfig(); err != nil {
		logger.Fatal("config: error reading config file: " + err.Error())
	}

	for _, key := range viper.AllKeys() {
		envKey := strings.ToUpper(strings.ReplaceAll(key, ".", "_"))
		err := viper.BindEnv(key, envKey)
		if err != nil {
			logger.Fatal("config: unable to bind env: " + err.Error())
		}
	}

	if err := viper.Unmarshal(&configs); err != nil {
		logger.Fatal("config: unable to decode into struct: " + err.Error())
	}
}

Do not use viper.AutomaticEnv().
From viper key database.host, create envKey DATABASE_HOST.
BindEnv: key => envKey

spf13/viper#188 (comment)

aeneasr mentioned this issue May 17, 2016

Decompositioning, implement Fosite ory/hydra#62

Merged

30 tasks

benoitmasson mentioned this issue Oct 6, 2016

Nested maps #195

Merged

benoitmasson mentioned this issue Oct 23, 2016

Fixed AllKeys() to include env values added with BindEnv() #264

Merged

natefinch mentioned this issue Jun 29, 2018

Unmarshal doesn't work with AutomaticEnv #522

Closed

martin-guthrie-docker mentioned this issue Sep 28, 2018

use viper unmarshal for environment variables martin-guthrie-docker/goclipkgtemplate#21

Open

sagikazarmark mentioned this issue Sep 12, 2019

Unmarshal non-bound environment variables #761

Closed

mudler mentioned this issue Apr 22, 2020

Allow to read config also from environment cloudfoundry-incubator/eirini-loggregator-bridge#14

Merged

FeodorFitsner mentioned this issue Feb 27, 2021

Enhancement: configure pglet server options via config file pglet/pglet#73

Closed

pierre referenced this issue in killbill/standalone-queue Jul 8, 2021

queue-server: Make the completion ack timeout configurable

1999ba3

vorlif mentioned this issue Oct 21, 2021

Unmarshal() does not use the env variable if the config file has no that key #1236

Closed

2 tasks

krakowski mentioned this issue Sep 6, 2022

Implement viper.BindStruct for automatic unmarshalling from environment variables #1429

Merged

iamolegga mentioned this issue Nov 9, 2022

e.VIper.ReadInConfig run as part of Unmarshal clears merged in fields iamolegga/enviper#12

Closed

tommartensen mentioned this issue Apr 21, 2023

fix env override when no value is set in config file stackrox/release-registry#11

Merged

andrewnathanco mentioned this issue Nov 16, 2023

[BUG]: Figure out issue with marshalling from environment variables VIPER andrewnathanco/mural#59

Closed

sagikazarmark mentioned this issue Dec 5, 2023

Make Unmarshal work with AutomaticEnv #1699

Closed

sagikazarmark closed this as completed in #1429 Dec 6, 2023

thesammy2010 added a commit to thesammy2010/api.thesammy2010.com that referenced this issue Jan 29, 2024

fix: config: add viper.BindEnv for DATABASE_URL

2f3067b

spf13/viper#188 (comment)

thesammy2010 mentioned this issue Jan 29, 2024

fix: config: add viper.BindEnv for DATABASE_URL thesammy2010/api.thesammy2010.com#14

Merged

charithabandi mentioned this issue May 7, 2024

# Load env variables into KwildConfig kwilteam/kwil-db#705

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Consider env vars when unmarshalling #188

Consider env vars when unmarshalling #188

aeneasr commented May 17, 2016 •

edited

Loading

rybit commented Oct 22, 2016

aeneasr commented Oct 22, 2016

aeneasr commented Oct 22, 2016

benoitmasson commented Oct 22, 2016 •

edited

Loading

burdiyan commented Jan 4, 2018

burdiyan commented Jan 4, 2018 •

edited

Loading

from-nibly commented May 9, 2018

from-nibly commented May 11, 2018

aarondl commented Jun 22, 2018

from-nibly commented Jun 22, 2018

aarondl commented Jun 22, 2018

from-nibly commented Jun 22, 2018

krak3n commented Jun 25, 2018 •

edited

Loading

natefinch commented Jun 29, 2018

supershal commented Aug 15, 2018 •

edited

Loading

krak3n commented Aug 16, 2018

unmultimedio commented Sep 11, 2019

sagikazarmark commented Sep 12, 2019

tunhuit commented Jun 17, 2021

Kavit900 commented Jun 2, 2022

76creates commented Jun 28, 2022

wsgomes commented Oct 11, 2022

Consider env vars when unmarshalling #188

Consider env vars when unmarshalling #188

Comments

aeneasr commented May 17, 2016 • edited Loading

rybit commented Oct 22, 2016

aeneasr commented Oct 22, 2016

aeneasr commented Oct 22, 2016

benoitmasson commented Oct 22, 2016 • edited Loading

burdiyan commented Jan 4, 2018

burdiyan commented Jan 4, 2018 • edited Loading

from-nibly commented May 9, 2018

from-nibly commented May 11, 2018

aarondl commented Jun 22, 2018

from-nibly commented Jun 22, 2018

aarondl commented Jun 22, 2018

from-nibly commented Jun 22, 2018

krak3n commented Jun 25, 2018 • edited Loading

natefinch commented Jun 29, 2018

supershal commented Aug 15, 2018 • edited Loading

krak3n commented Aug 16, 2018

unmultimedio commented Sep 11, 2019

sagikazarmark commented Sep 12, 2019

tunhuit commented Jun 17, 2021

Kavit900 commented Jun 2, 2022

76creates commented Jun 28, 2022

wsgomes commented Oct 11, 2022

aeneasr commented May 17, 2016 •

edited

Loading

benoitmasson commented Oct 22, 2016 •

edited

Loading

burdiyan commented Jan 4, 2018 •

edited

Loading

krak3n commented Jun 25, 2018 •

edited

Loading

supershal commented Aug 15, 2018 •

edited

Loading