Storing values larger than (1 << 31) - 2 (2 GB)

[mitar]

Currently bbolt has a hard-coded limit on value size of 2 GB. Could this be increased or made configurable? It seems to me that this is just to support mmap on 32 bit systems and there is really no design limitation why not to support larger values?

On my system, I have:

$ grep -m1 "address sizes" /proc/cpuinfo 
address sizes	: 39 bits physical, 48 bits virtual

So a bit less than 2^47 (128 TiB) could be the max limit for mmap on my system.

[simon28082]

bbolt db use bucket, every bucket max 2G, badger only single max 2G, @mitar Did you find a better solution?

[ahrtr]

Storing too big value isn't efficient. AFAIK, other database system, e.g mysql, also have limitation on value size.

Do you have a real use case which needs to store a value bigger than 2GB?

[tjungblu]

regardless of efficiency, wouldn't there be a limit in Golang's slice sizes? You'd have to store it in a []byte somehow. In which case you might want to shard it across several keys anyway.

[ahrtr]

There is no plan to support configuring the the max value size.

Please reopen this issue if you have a strong justification.