Skip to content

Compact data serializer/packer written in Go, intended to produce a cross-language usable format.

License

Notifications You must be signed in to change notification settings

ikkerens/ikeapack

Repository files navigation

IkeaPack GoDoc Go Report Card codecov

Named IkeaPack because it compacts structs in a very compact and packed manner, and if you don't know how to reassemble it, it may just look like a random blob of parts. Just like ikea products!

(If anyone from ikea doesn't like me using their name, just reach out to me and I'll change it, no copyright/trademark infringement intended.)

This is a packed struct serializer that is mostly meant for a private project but was released as it may be useful to someone else.

Originally this package was made as an extension to binary.Read and binary.Write, but I soon found those functions didn't match my use case as they offered no support for strings nor compression.

Features

  • Caches types for faster calls to the same type
  • Compression support
  • Tread safe (the calls are, reading to the value is not)
  • Easy to implement in other languages
  • Supported types:
    • uint8 (and byte) up to uint64
    • int8 up to int64
    • float32 and float64
    • string
    • anything implementing the Packer/Unpacker interfaces
    • slices
    • structs

Format

  • All primitives are stored in big endian format
  • All slices are stored with a uint32 prefix indicating their length
  • Strings are stored with a uint32 prefix indicating their length
  • Compression blocks are stored using deflate (level 9) with a uint32 prefixing the size of the compressed data blob

Note about int/uint

The types int and uint are not supported because their actual sizes depend on the compiler architecture.
Instead, be explicit and use int32/int64/uint32/uint64.

Note about nil values

This lib will initialize nil values when Unpacking/Unmarshalling, however it will panic if a nil value is attempted to be Packed/Serialized. This is due to the fact that there is no safe way to distinguish nil pointers from zero values.
Be safe, don't try to serialize nil values.

Include in your project

import "github.com/ikkerens/ikeapack"

Usage

package main

import (
	"bytes"
	"log"

	"github.com/ikkerens/ikeapack"
)

type myBlob struct {
	A uint64  // all fields have to be exported
	B []byte  `ikea:"compress:9"` // this field will be packed and compressed, with flate level 5
	C subBlob // If you omit the level `ikea:"compress"`, level 9 will be assumed.
	D int32
}

type subBlob struct {
	D string
}

func main() {
	b := new(bytes.Buffer)
	blob := &myBlob{A: 1, B: []byte{1, 2, 3, 4}, C: subBlob{D: "test message"}}

	// Pack
	if err := ikea.Pack(b, blob); err != nil { // Write does not need a pointer, but it is recommended
		log.Fatalln(err)
	}

	// Unpack
	newBlob := new(myBlob)
	if err := ikea.Unpack(b, newBlob); err != nil { // Read *needs* a pointer, or it will panic
		log.Fatalln(err)
	}

	log.Printf("Successfully unpacked: %+v", newBlob)
}

Benchmarks

These benchmarks can be found in alecthomas's go serialization benchmarks. While not all benchmarks are included since not all dependencies could resolve, these give a good overview of the performance of this lib vs the others.
Note that this library does not have a focus on being the fastest in any way, as this was made to cover a specific use-case. But it does strive to be as fast as it can be.

These benchmarks were executed on a Dell laptop with an i7-8550U cpu and 16GB of ram.

BenchmarkIkeaMarshal-8                           3000000               505 ns/op              72 B/op          8 allocs/op
BenchmarkIkeaUnmarshal-8                         2000000               670 ns/op             160 B/op         11 allocs/op
BenchmarkJsonMarshal-8                            500000              3785 ns/op            1224 B/op          9 allocs/op
BenchmarkJsonUnmarshal-8                          300000              4412 ns/op             464 B/op          7 allocs/op
BenchmarkEasyJsonMarshal-8                       1000000              1559 ns/op             784 B/op          5 allocs/op
BenchmarkEasyJsonUnmarshal-8                     1000000              1363 ns/op             160 B/op          4 allocs/op
BenchmarkBsonMarshal-8                           1000000              1433 ns/op             392 B/op         10 allocs/op
BenchmarkBsonUnmarshal-8                         1000000              1928 ns/op             244 B/op         19 allocs/op
BenchmarkGobMarshal-8                            2000000               930 ns/op              48 B/op          2 allocs/op
BenchmarkGobUnmarshal-8                          2000000               943 ns/op             112 B/op          3 allocs/op
BenchmarkXdrMarshal-8                            1000000              1740 ns/op             456 B/op         21 allocs/op
BenchmarkXdrUnmarshal-8                          1000000              1449 ns/op             240 B/op         11 allocs/op
BenchmarkUgorjiCodecMsgpackMarshal-8             1000000              1141 ns/op             561 B/op          6 allocs/op
BenchmarkUgorjiCodecMsgpackUnmarshal-8           1000000              1349 ns/op             449 B/op          6 allocs/op
BenchmarkSerealMarshal-8                          500000              2680 ns/op             912 B/op         21 allocs/op
BenchmarkSerealUnmarshal-8                        500000              2943 ns/op            1008 B/op         34 allocs/op
BenchmarkBinaryMarshal-8                         1000000              1427 ns/op             334 B/op         20 allocs/op
BenchmarkBinaryUnmarshal-8                       1000000              1554 ns/op             336 B/op         22 allocs/op
BenchmarkHproseMarshal-8                         2000000               971 ns/op             479 B/op          8 allocs/op
BenchmarkHproseUnmarshal-8                       1000000              1140 ns/op             320 B/op         10 allocs/op
BenchmarkGoAvroMarshal-8                          500000              2561 ns/op            1030 B/op         31 allocs/op
BenchmarkGoAvroUnmarshal-8                        200000              6346 ns/op            3437 B/op         87 allocs/op
BenchmarkGoAvro2TextMarshal-8                     500000              2875 ns/op            1326 B/op         20 allocs/op
BenchmarkGoAvro2TextUnmarshal-8                   500000              2690 ns/op             807 B/op         34 allocs/op
BenchmarkGoAvro2BinaryMarshal-8                  2000000               916 ns/op             510 B/op         11 allocs/op
BenchmarkGoAvro2BinaryUnmarshal-8                2000000               979 ns/op             576 B/op         13 allocs/op
BenchmarkProtobufMarshal-8                       2000000               984 ns/op             200 B/op          7 allocs/op
BenchmarkProtobufUnmarshal-8                     2000000               827 ns/op             192 B/op         10 allocs/op

Below you will find some of the benchmarks that do not rely on Go's reflection at runtime, which creates a significant performance boost, but for diligence it's worth mentioning here regardless.

BenchmarkMsgpMarshal-8                          10000000               178 ns/op             128 B/op          1 allocs/op
BenchmarkMsgpUnmarshal-8                         5000000               340 ns/op             112 B/op          3 allocs/op
BenchmarkFlatBuffersMarshal-8                    5000000               341 ns/op               0 B/op          0 allocs/op
BenchmarkFlatBuffersUnmarshal-8                  5000000               249 ns/op             112 B/op          3 allocs/op
BenchmarkCapNProtoMarshal-8                      3000000               483 ns/op              56 B/op          2 allocs/op
BenchmarkCapNProtoUnmarshal-8                    3000000               438 ns/op             200 B/op          6 allocs/op
BenchmarkCapNProto2Marshal-8                     2000000               723 ns/op             244 B/op          3 allocs/op
BenchmarkCapNProto2Unmarshal-8                   1000000              1019 ns/op             320 B/op          6 allocs/op
BenchmarkGoprotobufMarshal-8                     3000000               396 ns/op              96 B/op          2 allocs/op
BenchmarkGoprotobufUnmarshal-8                   2000000               614 ns/op             200 B/op         10 allocs/op
BenchmarkGogoprotobufMarshal-8                  10000000               162 ns/op              64 B/op          1 allocs/op
BenchmarkGogoprotobufUnmarshal-8                10000000               223 ns/op              96 B/op          3 allocs/op
BenchmarkColferMarshal-8                        10000000               133 ns/op              64 B/op          1 allocs/op
BenchmarkColferUnmarshal-8                      10000000               187 ns/op             112 B/op          3 allocs/op
BenchmarkGencodeMarshal-8                       10000000               179 ns/op              80 B/op          2 allocs/op
BenchmarkGencodeUnmarshal-8                     10000000               202 ns/op             112 B/op          3 allocs/op
BenchmarkGencodeUnsafeMarshal-8                 20000000               102 ns/op              48 B/op          1 allocs/op
BenchmarkGencodeUnsafeUnmarshal-8               10000000               146 ns/op              96 B/op          3 allocs/op
BenchmarkXDR2Marshal-8                          10000000               162 ns/op              64 B/op          1 allocs/op
BenchmarkXDR2Unmarshal-8                        10000000               136 ns/op              32 B/op          2 allocs/op