Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow a better calculation for chunksize that is actually divisible by typesize #52

Closed
FrancescAlted opened this issue Jul 27, 2016 · 2 comments

Comments

@FrancescAlted
Copy link
Member

For example, when trying to compress an image file with 24bit depth:

$ ll 24bit.bpt 
-rwx------ 1 faltet faltet 1377618 jul 26 14:15 24bit.bpt*

$ blpk -f c -t 3 24bit.bpt 24bit-shuffle.blp
Traceback (most recent call last):
  File "/home/faltet/miniconda/bin/blpk", line 11, in <module>
    sys.exit(main())
  File "/home/faltet/miniconda/lib/python2.7/site-packages/bloscpack/cli.py", line 457, in main
    metadata_args=MetadataArgs())
  File "/home/faltet/miniconda/lib/python2.7/site-packages/bloscpack/file_io.py", line 465, in pack_file
    metadata_args=metadata_args)
  File "/home/faltet/miniconda/lib/python2.7/site-packages/bloscpack/abstract_io.py", line 127, in pack
    (double_pretty_size(chunk_size), blosc_args.typesize)
bloscpack.exceptions.ChunkSizeTypeSizeMismatch: chunk_size: '1.0M (1048576B)' is not divisible by typesize: '3'

whereas if we help bloscpack passing a chunksize (via -z):

$ blpk -f c -t 3 -z 1377618 24bit.bpt 24bit-shuffle.blp

$ ll 24bit-shuffle.blp
-rw-rw-r-- 1 faltet faltet 40976 jul 27 17:10 24bit-shuffle.blp

Adding a more adaptative chunksize calculation would save the user to have to pass the chunksize manually.

This example is based on: https://github.com/Cyan4973/zstd/issues/256

@esc
Copy link
Member

esc commented Jul 27, 2016

The chunk-size and typesizes are hardcoded defaults that don't depend on the size if the input. If the user changes those values, it is his or her responsibility to "get it right"? But then again, if you want this feature you should implement it, I guess.

@FrancescAlted
Copy link
Member Author

Indeed. This is just a friendly reminder about the convenience for this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants