Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Some optimizations for sending UDP messages #2

Merged
merged 9 commits into from
Jan 23, 2016

Conversation

dqminh
Copy link

@dqminh dqminh commented Jan 21, 2016

This adds:

  • benchmark for various cases writing UDP messages out
  • improved performances in general cases witha bytes pool, and add support for sending uncompressed UDP messages with CompressionType=CompressNone
  • add supports for RawExtra attributes in messages, for cases where all messages from a source share the same extra attributes, so we only need to encode once.

Following is the benchmark:

=> git rev-list --reverse 4f441fa..8b73a5c
Checking out: 8fa7145 add a benchmark test
=> git checkout --quiet 8fa7145f561628a942e8027ac75a3dec2ee8a846
=> go test -benchmem -bench .
PASS
BenchmarkWriteBestSpeed-2       2000        599531 ns/op     1508341 B/op        185 allocs/op
ok      github.com/Graylog2/go-gelf/gelf    1.306s
=> git reset --hard --quiet

Checking out: 9352186 use marshaljsonbuf
=> git checkout --quiet 93521869f558cb3c5884d4fae61a2e163e4b00c5
=> go test -benchmem -bench .
PASS
BenchmarkWriteBestSpeed-2       2000        562657 ns/op     1508197 B/op        183 allocs/op
ok      github.com/Graylog2/go-gelf/gelf    1.234s
=> git reset --hard --quiet

Checking out: 9f8a322 add bytes pool for json and gzip
=> git checkout --quiet 9f8a32256b3c0497981c9df03ad8ea34c72cbe36
=> go test -benchmem -bench .
PASS
BenchmarkWriteBestSpeed-2       2000        582404 ns/op     1507220 B/op        179 allocs/op
ok      github.com/Graylog2/go-gelf/gelf    1.271s
=> git reset --hard --quiet

Checking out: 4a04cd7 add a test for no compression
=> git checkout --quiet 4a04cd7c3b68827d89c3157dff28da8ac72e14cc
=> go test -benchmem -bench .
PASS
BenchmarkWriteBestSpeed-2           2000        731160 ns/op     1507866 B/op        178 allocs/op
BenchmarkWriteNoCompression-2      20000         73277 ns/op      116671 B/op        128 allocs/op
ok      github.com/Graylog2/go-gelf/gelf    3.755s
=> git reset --hard --quiet

Checking out: d80a2b2 add CompressNone support which disable compression with a magic 2 byte header
=> git checkout --quiet d80a2b2eaabf3d212d4ae674349d1480807adb11
=> go test -benchmem -bench .
PASS
BenchmarkWriteBestSpeed-2                           2000        772494 ns/op     1508722 B/op        180 allocs/op
BenchmarkWriteNoCompression-2                      20000         57902 ns/op      113603 B/op        122 allocs/op
BenchmarkWriteDisableCompressionCompletely-2       50000         22889 ns/op        1853 B/op         23 allocs/op
ok      github.com/Graylog2/go-gelf/gelf    5.668s
=> git reset --hard --quiet

Checking out: 67732da refactor MarshalJSONBuf to save 1 allocation
=> git checkout --quiet 67732da8fde0014412de32e0dc73547ced55e4e7
=> go test -benchmem -bench .
PASS
BenchmarkWriteBestSpeed-2                           2000        641034 ns/op     1507898 B/op        178 allocs/op
BenchmarkWriteNoCompression-2                      20000         84399 ns/op      113152 B/op        122 allocs/op
BenchmarkWriteDisableCompressionCompletely-2       50000         25330 ns/op        1853 B/op         23 allocs/op
ok      github.com/Graylog2/go-gelf/gelf    5.581s
=> git reset --hard --quiet

Checking out: 45fe8a5 add RawExtra for pre-encoded extra attributes
=> git checkout --quiet 45fe8a5e04d87902f13e0b273cf086f782769b69
=> go test -benchmem -bench .
PASS
BenchmarkWriteBestSpeed-2                               2000        659069 ns/op     1507418 B/op        173 allocs/op
BenchmarkWriteNoCompression-2                          20000         68501 ns/op      114307 B/op        123 allocs/op
BenchmarkWriteDisableCompressionCompletely-2          100000         22870 ns/op        1625 B/op         21 allocs/op
BenchmarkWriteDisableCompressionAndPreencodeExtra-2   100000         15038 ns/op         836 B/op          7 allocs/op
ok      github.com/Graylog2/go-gelf/gelf    7.648s
=> git reset --hard --quiet

Checking out: 8b73a5c refactor duplicated code for compressing gzip/zlip
=> git checkout --quiet 8b73a5c1c7fcb26e4e0d72f1ae9f4ef075569cd7
=> go test -benchmem -bench .
PASS
BenchmarkWriteBestSpeed-2                               2000        663248 ns/op     1507446 B/op        173 allocs/op
BenchmarkWriteNoCompression-2                          20000        201816 ns/op      116357 B/op        127 allocs/op
BenchmarkWriteDisableCompressionCompletely-2           50000         36753 ns/op        1613 B/op         21 allocs/op
BenchmarkWriteDisableCompressionAndPreencodeExtra-2   100000         15673 ns/op         836 B/op          7 allocs/op
ok      github.com/Graylog2/go-gelf/gelf    10.029s
=> git reset --hard --quiet
=> git checkout --quiet 8b73a5c
OK for all revisions!

Signed-off-by: Daniel Dao <[email protected]>
Signed-off-by: Daniel Dao <[email protected]>
Signed-off-by: Daniel Dao <[email protected]>
/go test -benchmem -benchtime=5s -bench .
BenchmarkWriteBestSpeed-2          10000            545086 ns/op 1507203 B/op        178 allocs/op

Signed-off-by: Daniel Dao <[email protected]>
this saves allocation fr marshaling of same extra attributes for every messages

Signed-off-by: Daniel Dao <[email protected]>
@ashishgandhi
Copy link

👍

mariussturm pushed a commit that referenced this pull request Jan 23, 2016
Some optimizations for sending UDP messages
@mariussturm mariussturm merged commit e1cf699 into Graylog2:master Jan 23, 2016
@mariussturm
Copy link

Awesome work, thanks a lot!

IKukhta pushed a commit to IKukhta/go-gelf that referenced this pull request Sep 22, 2023
fixed reader logic to support chunked messages
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants