-
Notifications
You must be signed in to change notification settings - Fork 2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add zstd minetest.compress support #12515
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -4910,6 +4910,8 @@ Utilities | |
-- the amount of data in mod storage is not constrained by | ||
-- the amount of RAM available. (5.7.0) | ||
mod_storage_on_disk = true, | ||
-- "zstd" method for compress/decompress (5.7.0) | ||
compress_zstd = true, | ||
} | ||
|
||
* `minetest.has_feature(arg)`: returns `boolean, missing_features` | ||
|
@@ -6370,11 +6372,15 @@ Misc. | |
* `method` is a string identifying the compression method to be used. | ||
* Supported compression methods: | ||
* Deflate (zlib): `"deflate"` | ||
* Zstandard: `"zstd"` | ||
* `...` indicates method-specific arguments. Currently defined arguments | ||
are: | ||
* Deflate: `level` - Compression level, `0`-`9` or `nil`. | ||
* Zstandard: `level` - Compression level. Integer or `nil`. Default `3`. | ||
Note any supported Zstandard compression level could be used here, | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The documentation should probably mention that Zstd beats other algorithms not on how well data is compressed, but on the speed of decompression with a wide range of compression options. If this is not noted, mod and game developers may assume that using zstd compression is always better. With default compression, zstd yields ssize-savings similar to zlib/deflate (it is worse up until some boundary, then gets better). For inputs up to at least 4 kilobytes and even (in my tests) up to several 100 kilobytes at default compression level, zstd is likely to compress textual and biinary data about 5% worse than zlib … unless a pre-trained dictionary is used, which this API does not seem to provide. This is a WONTFIX from the zstd devs, as they are not interested in small files, but mods developers often do want to compress small payloads (e.g. bitmaps, item meta, json).. Modders should also be warned that while increasing the Zstd compression level can make the compressed output smaller, but can make the zstd encoder take over 20 times or – at the highest compression levels – over 100 times as long as the default level makes it take. You can try this out yourself: Compressing a 94 kilobyte HTML page took <0.01s at default compression level on my machine, but took >1s at one of the highest zstd compression levels. |
||
but these are subject to change between Zstandard versions. | ||
* `minetest.decompress(compressed_data, method, ...)`: returns data | ||
* Decompress a string of data (using ZLib). | ||
* Decompress a string of data using the algorithm specified by `method`. | ||
* See documentation on `minetest.compress()` for supported compression | ||
methods. | ||
* `...` indicates method-specific arguments. Currently, no methods use this | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The documentation should probably mention that Zstd beats other algorithms not on how well data is compressed, but on the speed of decompression with a wide range of compression options. If this is not noted, mod and game developers may assume that using zstd compression is always better. With default compression, zstd yields ssize-savings similar to zlib/deflate (it is worse up until some boundary, then gets better).
For inputs up to at least 4 kilobytes and even (in my tests) up to several 100 kilobytes at default compression level, zstd is likely to compress textual and biinary data about 5% worse than zlib … unless a pre-trained dictionary is used, which this API does not seem to provide. This is a WONTFIX from the zstd devs, as they are not interested in small files, but mods developers often do want to compress small payloads (e.g. bitmaps, item meta, json)..
Modders should also be warned that while increasing the Zstd compression level can make the compressed output smaller, but can make the zstd encoder take over 20 times or – at the highest compression levels – over 100 times as long as the default level makes it take. You can try this out yourself: Compressing a 94 kilobyte HTML page took <0.01s at default compression level on my machine, but took >1s at one of the highest zstd compression levels.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
...Since when was
lua_api.txt
meant to contain a detailed in-depth comparison between compression formats?