-
Notifications
You must be signed in to change notification settings - Fork 485
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Config reloads are not done if only a secret has changed #6978
Comments
This issue has not had any activity in the past 30 days, so the |
What's wrong?
Secrets (e.g. passwords) in configs files are hidden during marshalling to yaml. They are displayed as
<secret>
, in order to not expose them in logs and UIs. This causes issues for a static mode feature that configs won't be reloaded if a reload was triggered but the config hasn't changed. When Agent checks if the config changed, it marshals the yaml but the secrets are displayed as<secret>
and it doesn't detect if they've changed.There is a
NoScrubbedSecretsHook
function which is often used in our codebase to prevent this problem. However, it doesn't aways work. We may need to compare the config structs instead of marshaling to yaml.This bug may be present for all of logs, metrics, and traces subsystems. Also,
NoScrubbedSecretsHook
currently doesn't handle OTel secrets. And I'm not sure if we can implement it in a way that it will catch all possible secret types that come up in the future.Steps to reproduce
Run a config like this:
Then change the password in the config file and reload it. Check if the config was really reloaded. Do a similar test for logs and traces.
To illustrate the issue with
NoScrubbedSecretsHook
, this test doesn't pass:The reason is the call to
MarshalConfig(&c, true)
in/static/metrics/instance/instance.go
. I can't think of a good way to propagate the preference to not marshal to<secret>
. Global variables won't work, since there could be more than one thread marshalling at any given time.System information
No response
Software version
No response
Configuration
No response
Logs
No response
The text was updated successfully, but these errors were encountered: