Missing conversion rule for bytea in postgres

[quambene]

Error message:

Can't run dispatcher: ConnectorX(NoConversionRule("ByteA(true)", "connectorx::destinations::arrow::typesystem::ArrowTypeSystem"))

Expected: BYTEA in postgresql should be converted to Vec<u8> in Rust

Is this a time-consuming fix? Happy to assist if this is the case.

[dovahcrow]

I think it should be one new line at https://github.com/sfu-db/connector-x/blob/main/connectorx/src/transports/postgres_arrow.rs#L53 and every other transports.

 { ByteA[Vec<u8>]   => LargeBinary[Vec<u8>]     | conversion auto }

These two types, ByteA and LargeBinary, are from these two files.

https://github.com/sfu-db/connector-x/blob/main/connectorx/src/sources/postgres/typesystem.rs#L27
https://github.com/sfu-db/connector-x/blob/main/connectorx/src/destinations/arrow/typesystem.rs#L32

[quambene]

Thanks, I will test in my local project and submit PR if it's working.

[wangxiaoying]

Hi @quambene , we are planning to fix this issue in our next milestone (release next week). It will be great if you can submit this PR by this week. It's also ok if you are busy with other tasks, please let us know and we will work on this then.

[quambene]

@wangxiaoying Thanks for the reminder, I will definitely check it out this week.

[quambene]

@wangxiaoying I'm afraid I need your support. Here is my feedback:

1.) You might want to consider to tag your releases similar to https://github.com/pola-rs/polars/tags, i.e., differentiating your rust and python releases. This would be useful for testing bugfixes (https://doc.rust-lang.org/cargo/reference/overriding-dependencies.html#testing-a-bugfix), where you can point to a specific release after PR is merged: connectorx = { git = "https://github.com/sfu-db/connector-x/connectorx", rev = "<optional git tag>" }.

2.) Using connectorx = { version = "0.2.2-alpha.5", features = ["src_postgres", "dst_arrow2"] } and polars-core = "0.16", I could fix this error

Can't run dispatcher: ConnectorX(NoConversionRule("ByteA(true)",
"connectorx::destinations::arrow::typesystem::ArrowTypeSystem"))

by adding { ByteA[Vec<u8>] => LargeBinary[Vec<u8>] | conversion auto } to postgres_arrow.rs and postgres_arrow2.rs.

However, I'm getting an error in polars now:

Invalid operation Cannot create polars series from LargeBinary type

which is pointing to https://github.com/pola-rs/polars/blob/4f81a47a306b55f4409a61b6e7fb2313c85d658f/polars/polars-core/src/series/from.rs, and indicates that implementation of LargeBinary type is missing in polars as well?!

You can reproduce like this:

use connectorx::{
    destinations::arrow2::Arrow2Destination,
    prelude::{Dispatcher, PostgresArrow2Transport},
    sources::postgres::{rewrite_tls_args, BinaryProtocol, PostgresSource},
    sql::CXQuery,
};
use polars_core::frame::DataFrame;
use postgres::NoTls;
use url::Url;

pub fn query_postgres(url: Url, query: &str) -> Result<DataFrame, anyhow::Error> {
    let (config, _tls) = rewrite_tls_args(&url)?;
    let source = PostgresSource::<BinaryProtocol, NoTls>::new(config, NoTls, 10)?;
    let mut destination = Arrow2Destination::new();
    let queries = &[CXQuery::naked(query)];
    let dispatcher = Dispatcher::<_, _, PostgresArrow2Transport<BinaryProtocol, NoTls>>::new(
        source,
        &mut destination,
        queries,
        None,
    );
    dispatcher.run()?;
    let df = destination.polars();
    Ok(df?)
}

where url and query are defined accordingly.

Full branch can be found here: https://github.com/quambene/pigeon-rs/tree/pigeon-1

Any ideas?

[wangxiaoying]

Hi @quambene ,

1. Thanks for the suggestion! We will do that in our future release.
2. I think you're right. I got the same error as yours. I try to produce arrow instead of polars by using let df = destination.arrow(); and it works. So the error is because polars does not implement the LargeBinary type.

Thanks for the quick response!

[quambene]

@wangxiaoying Ok, I will open an issue over at polars then. Or have you already opened one?

[wangxiaoying]

@quambene I haven't yet. Please feel free to do it.