File exporter is not exporting the data

[yasha145]

Component(s)

exporter/file, receiver/otlpjsonfile

Describe the issue you're reporting

I am using otlpjsonfile as receiver and file exporter as my exporter.
After doing all the configuration and running the config file, I am getting the below log messages on repeat on terminal:

2024-03-08T14:18:25.019+0530    debug   fileconsumer/file.go:137        matched files   {"kind": "receiver", "name": "otlpjsonfile", "data_type": "metrics", "component": "fileconsumer", "paths": ["/Users/Documents/jsonFile.json"]}
2024-03-08T14:18:25.020+0530    debug   fileconsumer/file.go:175        Consuming files{paths 1 0  [/Users/Documents/jsonFile.json]}        {"kind": "receiver", "name": "otlpjsonfile", "data_type": "metrics", "component": "fileconsumer"}
2024-03-08T14:18:25.021+0530    info    fileconsumer/file.go:264        Started watching file   {"kind": "receiver", "name": "otlpjsonfile", "data_type": "metrics", "component": "fileconsumer", "path": "/Users/Documents/jsonFile.json"}

But this is not exporting anything to the mentioned file location. I am stuck how to proceed with this?

[github-actions]

Pinging code owners:

• exporter/file: @atingchen
• receiver/otlpjsonfile: @djaglowski @atoulme

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[djaglowski]

As with any troubleshooting problem involving multiple components, I suggest trying to isolate the components from one another so that you can see clearly which one is causing the problem. I recommend trying the debug exporter in place of the file exporter. If this exports data then the file exporter may be the problem. Otherwise, it may not be getting any data to export, so would indicate the receiver.

[yasha145]

Thanks for responding, I will try doing this.
Also, I just wanted to confirm otlpjson receiver takes json file as input right? Can you please share a snippet how the input file should look?

[atoulme]

see https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/protocol/file-exporter.md

[yasha145]

the logger.info() is not printing anything to the terminal
I am using debug exporter, it has marhsalled the data but not printing in on the terminal

[atoulme]

Sorry, please post your config yaml? I am a bit confused as to what you reference.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• exporter/file: @atingchen
• receiver/otlpjsonfile: @djaglowski @atoulme

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[damemi]

Adding because I might be having the same issue. Running contrib v0.99.0, here is my config:

exporters:
  debug:
    verbosity: detailed
  file:
    path: /home/mikedame/test/fixtures/output.json
processors:

receivers:
  otlpjsonfile:
    poll_interval: 5s
    replay_file: true
    start_at: beginning
    include:
      - "/home/mikedame/test/fixtures/*.json"
service:
  pipelines:
    traces:
      exporters:
        - file
        - debug
      processors:

      receivers:
        - otlpjsonfile

Logs hang at watching file:

$ ./otelcontribcol_linux_amd64 --config test/generated_collector.yaml
2024-05-28T17:08:02.144Z	info	[email protected]/service.go:99	Setting up own telemetry...
2024-05-28T17:08:02.144Z	info	[email protected]/telemetry.go:103	Serving metrics	{"address": ":8888", "level": "Normal"}
2024-05-28T17:08:02.144Z	info	[email protected]/exporter.go:275	Development component. May change in the future.	{"kind": "exporter", "data_type": "traces", "name": "debug"}
2024-05-28T17:08:02.146Z	info	[email protected]/service.go:166	Starting otelcontribcol...	{"Version": "0.99.0-dev", "NumCPU": 24}
2024-05-28T17:08:02.147Z	info	extensions/extensions.go:34	Starting extensions...
2024-05-28T17:08:02.163Z	info	[email protected]/service.go:192	Everything is ready. Begin running and processing data.
2024-05-28T17:08:02.163Z	warn	localhostgate/featuregate.go:63	The default endpoints for all servers in components will change to use localhost instead of 0.0.0.0 in a future version. Use the feature gate to preview the new default.	{"feature gate ID": "component.UseLocalHostAsDefaultHost"}
2024-05-28T17:08:02.364Z	info	fileconsumer/file.go:231	Started watching file	{"kind": "receiver", "name": "otlpjsonfile", "data_type": "traces", "component": "fileconsumer", "path": "/home/mikedame/test/fixtures/input.json"}

I'm using this OTLP JSON trace file as input, generated with the file exporter.

I added start_at, replay_file, and poll_interval thinking this could be because it's just waiting for new data to be written to the file (similar to #31750 (comment)), but those settings haven't helped.

[atoulme]

That file was definitely manipulated. The format is JSONL. You prettified the traces and now they can't be read back anymore.

[damemi]

Ah thanks @atoulme, that was the issue! You're right, I did prettify the file exporter output with jq and didn't realize that would have an effect in this case. I ran the file through jq -c to remove the line breaks and now I can see the traces being read and exported.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• exporter/file: @atingchen
• receiver/otlpjsonfile: @djaglowski @atoulme

See Adding Labels via Comments if you do not have permissions to add labels yourself.