vector protobuf encoding uses 3x CPU compared to JSON encoding

[nmiculinic]

Image: timberio/vector:0.42.0-debian

I was very surprised when I switch kafka encoding to protobuf from JSON having 4.5x increase in CPU usage. Config:

  kafka:
    compression: gzip
    encoding:
      codec: protobuf
      protobuf:
        desc_file: proto.desc
        message_type: <>
    batch:
      max_bytes: 1048576 # 1MiB
      max_events: 100
      timeout_secs: 1

with proto being (simple string key/value fields):

message LogRecord {
  ...

  // log message. 
  string message = 3;
  // log level (e.g. info, warn, error, debug)
  string level = 4;

  
   ...

  // timestamp in unix_ns
    google.protobuf.Timestamp timestamp = 24;
}

So yeah, I'm not sure what's happening here, or how to get performance in line with plain JSON shemaless encoding

[nmiculinic]

Fwiw I had issue getting timestamp in some format vector can serialize by default, thus why I've used unix_ns uint64.

[pront]

Hi @nmiculinic, this is surprising indeed. Can you use your favorite profiler so we get some insights?

https://github.com/flamegraph-rs/flamegraph is one option.

I also want to ensure we compare meaningful numbers, e.g. ensure we are using release builds.

---

I had issue getting timestamp in some format vector can serialize by default, thus why I've used unix_ns uint64

We can discuss this in another thread to avoid over-complicating this one.

[nmiculinic]

I've figured out the timestamp part, with import "google/protobuf/timestamp.proto";
Let me see what can I do about flamegraph

[nmiculinic]

[nmiculinic]

I've used https://github.com/brendangregg/FlameGraph

hmm.... I can only have perf on the machine which is problematic. No way can I install arbitrary binary there from random rust codebase.

Perf command (maybe I've made a mistake)?:

perf record -p <PID> -o perf.data -g -F 99

scp data to my machine, and then

stackcollapse-perf.pl out.perf > out.folded
flamegraph.pl out.folded > flamegraph.svg

This is running with containerd I think, timberio/vector:0.42.0-debian image. I can change to some other docker image which is compiled with more information.

[nmiculinic]

Fwiw I only see performance issues on the sink/write side of things.

read/source side when decoding from kafka doesn't seem to suffer same issues

[pront]

Thank you for providing all this so quickly. I think we will need a more fine-grained of what's happening on the encoding side to understand if this is something we can easily improve on. We rely on an internal framework for performance analysis but we didn't add any protobuf experiment. If you are interested, the existing cases are here: https://github.com/vectordotdev/vector/tree/master/regression/cases

[nmiculinic]

Ok, let me know if I can help somehow. Seems like you need to add protobuf experiment there and analyse it in depth for performance issues?

[pront]

Seems like you need to add protobuf experiment there and analyse it in depth for performance issues?

Yes, we can use that to establish a baseline and evaluate future performance enhancements.