Configure the javaagent to log to a file (instead of stderr)

[mateuszrzeszutek]

Is your feature request related to a problem? Please describe.
The aim of this issue is to add a new otel.javaagent.log-file (or something similar) property - when configured, the agent will use that file as the log output instead of logging everything to standard error.
Some users want to keep the agent logs separate from application logs; and with debug mode enabled I suppose that it can be more useful to have all the agent logs in a separate file, makes it easier to file a bug.

Additional context
You can already do that by setting io.opentelemetry.javaagent.slf4j.simpleLogger.logFile system property, but I guess that it'd be much nicer if we supported our own property (and env variable) and officially documented it.

[trask]

logging to a file that doesn't roll over worries me

[anuraaga]

I think we've talked about logging before so I may just be forgetting a nuance. Is there any way we could make a best effort attempt to redirect our logs into the app's slf4j when it's available? Perhaps we can add a transformer to slf4j which statically registers it into some interface we provide in the bootstrap classloader. Having agent logs be configurable like the app would be so nice.

[trask]

I think that starts to run into similar issues as using app's otel api/sdk. I think there's interest in solving those issues for otel api/sdk (#3261), which could lead to solution for slf4j too.

[mikeweisskopf]

I think a related issue comes with the use of logging libraries like logback in the actual application being instrumented. If you have logback configured to do application logging to a rotating set of files (using a RollingFileAppender) and run the application in a docker container then the javaagent logs will end up in the docker container logs (because the agent is logging to standard error) while all the rest of the application logs are in files managed by logback. We don't really want to have to manage logs separately (and remember where to look for what), but its not clear how to wire the two together as the agent starts up before the application (and its logback bootstrap).

[anuraaga]

I think that starts to run into similar issues as using app's otel api/sdk

The concept is definitely similar but I think logging should be much simpler though without needing a highly complex bridge given the API surface is much smaller (just ~4 methods, getLogger, warn, info, debug) and we would not be creating any types, just using stuff - one of the big complexities of otel api bridging is the need to create types in the agent for use in the app and vice versa, but nothing like that for logging. So basically just use reflection to access the app's slf4j version of those methods if no reflective exception or fallback to our shaded one that uses simplelogger. ClassLoader instrumentation can register a handle to the app's slf4j methods into the bootstrap classloader, somewhat like what we do for forceFlush. Or if we know how to add a static initializer with instrumentation, we could potentially avoid reflection with slf4j instrumentation that adds a static initializer to LoggerFactory which registers itself into the bootstrap classloader

[trask]

ya, the part I think is tricky, is what to do in application server environments with multiple applications / copies of slf4j, though since this behavior would be opt-in, we can just document that restriction 👍

[mateuszrzeszutek]

Or if we know how to add a static initializer with instrumentation, we could potentially avoid reflection with slf4j instrumentation that adds a static initializer to LoggerFactory which registers itself into the bootstrap classloader

Even if we did that, we could only register the "our slf4j" to "app slf4j" bridge after the agent has initialized and all instrumentations have been installed - everything that was logged earlier would still go to stderr.

[anuraaga]

everything that was logged earlier would still go to stderr.

Yep - in that sense, using reflection with classloader instrumentation does seem a bit better since it would reduce the number of messages that may end up missed. If we can get log messages printed during instrumentation of classes, and for LOGGING exporters into the app's slf4j that would be pretty good coverage.

[johnbley]

Additionally / alternately, is there / should there be a way to completely suppress our usage of stderr (and just be silent, even about critical errors)?

[trask]

Additionally / alternately, is there / should there be a way to completely suppress our usage of stderr (and just be silent, even about critical errors)?

For now at least, try -Dio.opentelemetry.javaagent.slf4j.simpleLogger.defaultLogLevel=off

[zmapleshine]

Logs generated by the agent itself can sometimes be valuable, for example by printing out ’Timeout‘ or "Error" logs for a component and associating them with traceID to consume and clean them using something like Flink. The application itself doesn't care about these logs, so I think the agent's logs do need to be exported to a separate path.

[mateuszrzeszutek]

Additionally / alternately, is there / should there be a way to completely suppress our usage of stderr (and just be silent, even about critical errors)?

Hmm, something like -Dotel.javaagent.no-logging=true? That would be a good thing to have too.

[jkwatson]

logging to a file that doesn't roll over worries me

Whatever is built, it should have rolling logs built in as the default.

[trask]

Hmm, something like -Dotel.javaagent.no-logging=true? That would be a good thing to have too.

Maybe -Dotel.javaagent.logging.level=debug/off/etc and -Dotel.javaagent.logging.file=... or similar?

[XiaoWeiKIN]

mark

[danhngo-lx]

For now at least, try -Dio.opentelemetry.javaagent.slf4j.simpleLogger.defaultLogLevel=off

Hi @trask, as I see we have switched to jul instead of slf4j, is this still valid? Or what is the right way to suppress the logs of the javaagent now?

[mateuszrzeszutek]

Hi @danhngo-lx ,

Hi @trask, as I see we have switched to jul instead of slf4j, is this still valid? Or what is the right way to suppress the logs of the javaagent now?

Yes, it is still valid -- we're still using slf4j internally as the actual logging implementation (we're simply redirecting all those JUL calls).