Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support Parquet BYTE_STREAM_SPLIT for INT32, INT64, and FIXED_LEN_BYTE_ARRAY primitive types #6159

Merged
merged 24 commits into from
Aug 6, 2024
Merged

Support Parquet BYTE_STREAM_SPLIT for INT32, INT64, and FIXED_LEN_BYTE_ARRAY primitive types #6159

merged 24 commits into from
Aug 6, 2024

Commits on Jul 30, 2024

  1. add todos to help trace flow

    @etseidl
    etseidl committed Jul 30, 2024
    Configuration menu
    Copy the full SHA
    71ae965 View commit details
    Browse the repository at this point in the history

  2. add support for byte_stream_split encoding for INT32 and INT64 data

    @etseidl
    etseidl committed Jul 30, 2024
    Configuration menu
    Copy the full SHA
    dddf8a0 View commit details
    Browse the repository at this point in the history

  3. byte_stream_split encoding for fixed_len_byte_array

    @etseidl
    etseidl committed Jul 30, 2024
    Configuration menu
    Copy the full SHA
    1c1af32 View commit details
    Browse the repository at this point in the history

  4. revert changes to Decoder and add VariableWidthByteStreamSplitDecoder

    @etseidl
    etseidl committed Jul 30, 2024
    Configuration menu
    Copy the full SHA
    75ea319 View commit details
    Browse the repository at this point in the history

  5. remove set_type_width as it is now unused

    @etseidl
    etseidl committed Jul 30, 2024
    Configuration menu
    Copy the full SHA
    7ce40ae View commit details
    Browse the repository at this point in the history

  6. begin implementing roundtrip test

    @etseidl
    etseidl committed Jul 30, 2024
    Configuration menu
    Copy the full SHA
    e7829c3 View commit details
    Browse the repository at this point in the history

  7. move test

    @etseidl
    etseidl committed Jul 30, 2024
    Configuration menu
    Copy the full SHA
    c0eb828 View commit details
    Browse the repository at this point in the history

Commits on Aug 1, 2024

  1. clean up some documentation

    @etseidl
    etseidl committed Aug 1, 2024
    Configuration menu
    Copy the full SHA
    fec8001 View commit details
    Browse the repository at this point in the history

  2. add test of byte_stream_split with flba

    @etseidl
    etseidl committed Aug 1, 2024
    Configuration menu
    Copy the full SHA
    b9d4baf View commit details
    Browse the repository at this point in the history

  3. add check for and test of mismatched sizes

    @etseidl
    etseidl committed Aug 1, 2024
    Configuration menu
    Copy the full SHA
    f8ee320 View commit details
    Browse the repository at this point in the history

  4. remove type_length from Encoder and add VaribleWidthByteStreamSplitEn… 

    …coder
    @etseidl
    etseidl committed Aug 1, 2024
    Configuration menu
    Copy the full SHA
    29c5119 View commit details
    Browse the repository at this point in the history

  5. Merge remote-tracking branch 'origin/master' into bss

    @etseidl
    etseidl committed Aug 1, 2024
    Configuration menu
    Copy the full SHA
    3e598be View commit details
    Browse the repository at this point in the history

  6. fix clippy error

    @etseidl
    etseidl committed Aug 1, 2024
    Configuration menu
    Copy the full SHA
    ef14c7d View commit details
    Browse the repository at this point in the history

  7. change type of argument to new()

    @etseidl
    etseidl committed Aug 1, 2024
    Configuration menu
    Copy the full SHA
    6513ffb View commit details
    Browse the repository at this point in the history

  8. formatting

    @etseidl
    etseidl committed Aug 1, 2024
    Configuration menu
    Copy the full SHA
    3a650a7 View commit details
    Browse the repository at this point in the history

  9. add another test

    @etseidl
    etseidl committed Aug 1, 2024
    Configuration menu
    Copy the full SHA
    c63a1ce View commit details
    Browse the repository at this point in the history

Commits on Aug 2, 2024

  1. add variable to split/join streams for FLBA

    @etseidl
    etseidl committed Aug 2, 2024
    Configuration menu
    Copy the full SHA
    09f467d View commit details
    Browse the repository at this point in the history

  2. more informative error message

    @etseidl
    etseidl committed Aug 2, 2024
    Configuration menu
    Copy the full SHA
    3fd6bc5 View commit details
    Browse the repository at this point in the history

  3. avoid buffer copies in decoder per suggestion from review

    @etseidl
    etseidl committed Aug 2, 2024
    Configuration menu
    Copy the full SHA
    c6bb2ef View commit details
    Browse the repository at this point in the history

Commits on Aug 5, 2024

  1. add roundtrip test

    @etseidl
    etseidl committed Aug 5, 2024
    Configuration menu
    Copy the full SHA
    3f6d944 View commit details
    Browse the repository at this point in the history

  2. optimized version...but clippy complains

    @etseidl
    etseidl committed Aug 5, 2024
    Configuration menu
    Copy the full SHA
    97d159b View commit details
    Browse the repository at this point in the history

  3. clippy was right...replace loop with copy_from_slice

    @etseidl
    etseidl committed Aug 5, 2024
    Configuration menu
    Copy the full SHA
    340eab4 View commit details
    Browse the repository at this point in the history

  4. fix test

    @etseidl
    etseidl committed Aug 5, 2024
    Configuration menu
    Copy the full SHA
    b2d90ce View commit details
    Browse the repository at this point in the history

  5. optimize split_streams_variable for long type widths

    @etseidl
    etseidl committed Aug 5, 2024
    Configuration menu
    Copy the full SHA
    104b72e View commit details
    Browse the repository at this point in the history