Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IPC reader fails to decode a struct array with zero fields #4643

Closed
kawadakk opened this issue Aug 4, 2023 · 4 comments · Fixed by #5918
Closed

IPC reader fails to decode a struct array with zero fields #4643

kawadakk opened this issue Aug 4, 2023 · 4 comments · Fixed by #5918
Labels

Comments

@kawadakk
Copy link
Contributor

kawadakk commented Aug 4, 2023

Describe the bug
Encoding succeeds, but decoding fails with an error InvalidArgumentError("all columns in a record batch must have the specified row count").

To Reproduce

// add to arrow-ipc/src/writer.rs
#[test]
fn ipc_struct_array_with_zero_fields() {
    let record_batch = {
        let struct_ty = DataType::Struct(Vec::<Field>::new().into());
        let structs = ArrayDataBuilder::new(struct_ty.clone())
            .len(3)
            .build()
            .unwrap();
        RecordBatch::try_new_with_options(
            Arc::new(Schema::new(vec![Field::new("field", struct_ty, false)])),
            vec![make_array(structs)],
            &RecordBatchOptions::new().with_row_count(Some(3)),
        )
        .unwrap()
    };
    let deserialized_batch = deserialize(serialize(&record_batch));

    let structs = deserialized_batch
        .column(0)
        .as_any()
        .downcast_ref::<StructArray>()
        .unwrap();

    assert_eq!(structs.len(), 3);
}
thread 'writer::tests::ipc_struct_array_with_zero_fields' panicked at 'called `Result::unwrap()` on an `Err` value: InvalidArgumentError("all columns in a record batch must have the specified row count")', arrow-ipc\src\writer.rs:1816:39
stack backtrace:
   0: rust_begin_unwind
             at /rustc/864bdf7843e1ceabc824ed86d97006acad6af643/library\std\src/panicking.rs:617:5
   1: core::panicking::panic_fmt
             at /rustc/864bdf7843e1ceabc824ed86d97006acad6af643/library\core\src/panicking.rs:67:14
   2: core::result::unwrap_failed
             at /rustc/864bdf7843e1ceabc824ed86d97006acad6af643/library\core\src/result.rs:1651:5
   3: core::result::Result<T,E>::unwrap
             at /rustc/864bdf7843e1ceabc824ed86d97006acad6af643\library\core\src/result.rs:1076:23
   4: arrow_ipc::writer::tests::deserialize
             at .\src\writer.rs:1816:9
   5: arrow_ipc::writer::tests::ipc_struct_array_with_zero_fields
             at .\src\writer.rs:1976:34
   6: arrow_ipc::writer::tests::ipc_struct_array_with_zero_fields::{{closure}}
             at .\src\writer.rs:1962:44
   7: core::ops::function::FnOnce::call_once
             at /rustc/864bdf7843e1ceabc824ed86d97006acad6af643\library\core\src\ops/function.rs:250:5
   8: core::ops::function::FnOnce::call_once
             at /rustc/864bdf7843e1ceabc824ed86d97006acad6af643/library\core\src\ops/function.rs:250:5

Expected behavior
Successful roundtrip conversion

Additional context

@kawadakk kawadakk added the bug label Aug 4, 2023
@jackwener
Copy link
Member

When deserialize

fn try_new_impl(
        schema: SchemaRef,
        columns: Vec<ArrayRef>,
        options: &RecordBatchOptions,
    )

column in columns len is 0.
options.rowcount is 3

@hiltontj
Copy link
Contributor

Should the expected behaviour be that it fails to encode the data in the first place? (instead of roundtrip conversion)

Or, is it possible for the reader to decode the struct array without knowledge of what fields it has?

@kawadakk
Copy link
Contributor Author

If it can decode a struct array with a non-zero number of fields, it should be able to decode one with zero fields as well.

@kawadakk
Copy link
Contributor Author

I opened a PR: #5918

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants