Cannot create polars series from LargeBinary type

[quambene]

Are you using Python or Rust?

Rust

Which feature gates did you use?

None

What version of polars are you using?

polars = "0.16" and polars-core = "0.16"

What operating system are you using polars on?

Ubuntu

Describe your bug.

If you are querying bytea data from a postgres database, the following error occurs in polars:

Invalid operation Cannot create polars series from LargeBinary type

What are the steps to reproduce the behavior?

Using connectorx = { version = "0.2.2-alpha.5", features = ["src_postgres", "dst_arrow2"] }, polars-core = "0.16", postgres = "0.19.2", and url = "2.2.2", you can reproduce as follows:

use connectorx::{
    destinations::arrow2::Arrow2Destination,
    prelude::{Dispatcher, PostgresArrow2Transport},
    sources::postgres::{rewrite_tls_args, BinaryProtocol, PostgresSource},
    sql::CXQuery,
};
use polars_core::frame::DataFrame;
use postgres::NoTls;
use url::Url;

pub fn query_postgres(url: Url, query: &str) -> Result<DataFrame, anyhow::Error> {
    let (config, _tls) = rewrite_tls_args(&url)?;
    let source = PostgresSource::<BinaryProtocol, NoTls>::new(config, NoTls, 10)?;
    let mut destination = Arrow2Destination::new();
    let queries = &[CXQuery::naked(query)];
    let dispatcher = Dispatcher::<_, _, PostgresArrow2Transport<BinaryProtocol, NoTls>>::new(
        source,
        &mut destination,
        queries,
        None,
    );
    dispatcher.run()?;
    let df = destination.polars();
    Ok(df?)
}

where url = Url::parse("postgresql:https://db_user:db_password@db_host:db_port/db_name") and the query should query data which includes bytea type in postgres.

What is the actual behavior?

The error output is

Invalid operation Cannot create polars series from LargeBinary type

What is the expected behavior?

Bytea type in postgres converts to LargeBinary type in arrow2. However, LargeBinary type is missing in polars.

If you need anything else or I can assist, please let me know.

Linking downstream issues for completeness: sfu-db/connector-x#148 and quambene/pigeon-rs#1.

[ritchie46]

Polars does not support the binary data type. Maybe you can convert to type List<u8>?

[quambene]

I'm not sure. Maybe @wangxiaoying can help out if this might be possible in connectorx?

[ritchie46]

I'm not sure. Maybe @wangxiaoying can help out if this might be possible in connectorx?

No, this is not related to connectorx. We could do the conversion in polars as well. I will take a look.

[quambene]

@ritchie46 That would be great.

Just a remark if you need to use the steps to reproduce above. Version 0.2.2-alpha.5 of connectorx is missing on crates.io and I used a [patch.crates-io] in my Cargo.toml including a local fix for connectorx. But I assume, you won't need it anyway.