-
Notifications
You must be signed in to change notification settings - Fork 27
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
dietgpu float32 compression support: part 1 (#8)
Summary: Pull Request resolved: #8 This diff contains some prerequisites for losslessly compressing float32 data. We can now accept float32 dtype data, but right now the kernels that pack the uncompressed portion have not been adjusted to allow storing 3 bytes of data per word, so instead we store the uncompressed 3 bytes as a 4 byte word. As a result, this diff does not actually compress float32 data, but instead expands it. A subsequent diff will either allow for the split/join float kernel to handle 3 byte words, or perhaps it will be more efficient to allow interleaving 2 ANS compressors in the same data stream using different table data, which would allow compressing 2 byte words of data with statistics independently computed for each byte. This diff also removes the "min symbol" usage from the compressor and decompressor. The original idea for this was to perhaps allow for optimizations when there are <= 32 or 64 symbols by storing table data in registers, but this proved impractical. Additionally only the symbols that are present need be stored in the ANS header data, but entries for all 256 possible symbols are currently stored. We still compute the min and num symbols which can bound the data being stored, but we aren't doing anything with them. A subsequent diff might allow for storing a truncated symbol table if all 256 possible symbols are not in fact encountered. Removing this min symbol usage provides a small speedup and will make it easier to interleave 2 ANS compressors in the same stream by removing one less thing that needs to be handled. Reviewed By: jspark1105 Differential Revision: D34157690 fbshipit-source-id: 64e2c8d6f5ab3acdb3441ed62717cb1638fb21e3
- Loading branch information
1 parent
d089eaa
commit 8f95d1c
Showing
11 changed files
with
544 additions
and
380 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.