object-store fails to compile for `wasm32-unknown-unknown` with `http` feature #4776

kylebarron · 2023-09-05T02:48:07Z

Describe the bug

object-store crate fails to compile for wasm32 with the http feature.

There are a ton of errors. Some unresolved imports, some future is not `Send` , etc.

error[E0432]: unresolved imports `reqwest::NoProxy`, `reqwest::Proxy`
   --> src/client/mod.rs:51:38
    |
51  | use reqwest::{Client, ClientBuilder, NoProxy, Proxy, RequestBuilder};
    |                                      ^^^^^^^  ^^^^^ no `Proxy` in the root
    |                                      |
    |                                      no `NoProxy` in the root
    |
note: found an item that was configured out
   --> /Users/kyle/.cargo/registry/src/index.crates.io-6f17d22bba15001f/reqwest-0.11.20/src/lib.rs:322:33
    |
322 |     pub use self::proxy::{Proxy,NoProxy};
    |                                 ^^^^^^^
note: found an item that was configured out
   --> /Users/kyle/.cargo/registry/src/index.crates.io-6f17d22bba15001f/reqwest-0.11.20/src/lib.rs:322:27
    |
322 |     pub use self::proxy::{Proxy,NoProxy};
    |                           ^^^^^

error[E0432]: unresolved import `crate::client::header`
  --> src/http/mod.rs:43:20
   |
43 | use crate::client::header::header_meta;
   |                    ^^^^^^ could not find `header` in `client`

error[E0433]: failed to resolve: could not find `header` in `client`
  --> src/http/mod.rs:66:32
   |
66 |         source: crate::client::header::Error,
   |                                ^^^^^^ could not find `header` in `client`

error[E0433]: failed to resolve: could not find `tls` in `reqwest`
   --> src/client/mod.rs:465:30
    |
465 |                     reqwest::tls::Certificate::from_pem(certificate.as_bytes())
    |                              ^^^ could not find `tls` in `reqwest`

warning: unused macro definition: `builder_opts`
   --> src/parse.rs:108:14
    |
108 | macro_rules! builder_opts {
    |              ^^^^^^^^^^^^
    |
    = note: `#[warn(unused_macros)]` on by default

error: future cannot be sent between threads safely
   --> src/http/mod.rs:98:70
    |
98  |       async fn put(&self, location: &Path, bytes: Bytes) -> Result<()> {
    |  ______________________________________________________________________^
99  | |         self.client.put(location, bytes).await
100 | |     }
    | |_____^ future created by async block is not `Send`
    |
    = help: within `[async block@src/http/mod.rs:98:70: 100:6]`, the trait `std::marker::Send` is not implemented for `*mut u8`
note: future is not `Send` as this value is used across an await
   --> src/http/client.rs:172:66
    |
166 |             match builder.send_retry(&self.retry_config).await {
    |                   -------------------------------------------- has type `std::result::Result<reqwest::Response, retry::Error>` which is not `Send`
...
172 |                         self.create_parent_directories(location).await?
    |                                                                  ^^^^^ await occurs here, with `builder.send_retry(&self.retry_config).await` maybe used later
...
177 |         }
    |         - `builder.send_retry(&self.retry_config).await` is later dropped here
    = note: required for the cast from `Pin<Box<[async block@src/http/mod.rs:98:70: 100:6]>>` to `Pin<Box<dyn Future<Output = Result<(), Error>> + Send>>`
    = note: the full name for the target type has been written to '/Users/kyle/github/apache/arrow-rs/object_store/target/wasm32-unknown-unknown/debug/deps/object_store-87363e35aacc8eb7.long-type-9982652401887617439.txt'

error[E0277]: `Rc<RefCell<wasm_bindgen_futures::Inner>>` cannot be sent between threads safely
    --> src/http/mod.rs:125:14
     |
125  |             .boxed();
     |              ^^^^^ `Rc<RefCell<wasm_bindgen_futures::Inner>>` cannot be sent between threads safely
     |
     = help: within `futures::stream::Map<wasm_streams::readable::into_stream::IntoStream<'static>, [closure@reqwest::Response::bytes_stream::{closure#0}]>`, the trait `std::marker::Send` is not implemented for `Rc<RefCell<wasm_bindgen_futures::Inner>>`
note: required because it appears within the type `JsFuture`
    --> /Users/kyle/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasm-bindgen-futures-0.4.37/src/lib.rs:98:12
     |
98   | pub struct JsFuture {
     |            ^^^^^^^^
note: required because it appears within the type `Option<JsFuture>`
    --> /Users/kyle/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/option.rs:563:10
     |
563  | pub enum Option<T> {
     |          ^^^^^^
note: required because it appears within the type `IntoStream<'static>`
    --> /Users/kyle/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasm-streams-0.3.0/src/readable/into_stream.rs:22:12
     |
22   | pub struct IntoStream<'reader> {
     |            ^^^^^^^^^^
note: required because it appears within the type `Map<IntoStream<'static>, [closure@Response::bytes_stream::{closure#0}]>`
    --> /Users/kyle/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/stream/map.rs:15:16
     |
15   |     pub struct Map<St, F> {
     |                ^^^
     = note: required for `Unique<Map<IntoStream<'static>, ...>>` to implement `std::marker::Send`
     = note: the full type name has been written to '/Users/kyle/github/apache/arrow-rs/object_store/target/wasm32-unknown-unknown/debug/deps/object_store-87363e35aacc8eb7.long-type-262376943904190086.txt'
note: required because it appears within the type `Box<Map<IntoStream<'static>, [closure@Response::bytes_stream::{closure#0}]>>`
    --> /Users/kyle/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/alloc/src/boxed.rs:195:12
     |
195  | pub struct Box<
     |            ^^^
note: required because it appears within the type `Pin<Box<Map<IntoStream<'static>, [closure@Response::bytes_stream::{closure#0}]>>>`
    --> /Users/kyle/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/pin.rs:410:12
     |
410  | pub struct Pin<P> {
     |            ^^^
note: required because it appears within the type `impl Stream<Item = Result<Bytes, Error>>`
    --> /Users/kyle/.cargo/registry/src/index.crates.io-6f17d22bba15001f/reqwest-0.11.20/src/wasm/response.rs:137:34
     |
137  |     pub fn bytes_stream(self) -> impl futures_core::Stream<Item = crate::Result<Bytes>> {
     |                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: required because it appears within the type `IntoStream<impl Stream<Item = Result<Bytes, Error>>>`
    --> /Users/kyle/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/try_stream/into_stream.rs:12:16
     |
12   |     pub struct IntoStream<St> {
     |                ^^^^^^^^^^
note: required because it appears within the type `Map<IntoStream<impl Stream<Item = Result<Bytes, Error>>>, MapErrFn<[[email protected]:124:22]>>`
    --> /Users/kyle/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/stream/map.rs:15:16
     |
15   |     pub struct Map<St, F> {
     |                ^^^
note: required because it appears within the type `MapErr<impl Stream<Item = Result<Bytes, Error>>, [[email protected]:124:22]>`
    --> /Users/kyle/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/try_stream/mod.rs:63:5
     |
63   |     MapErr<St, F>(
     |     ^^^^^^
note: required by a bound in `futures::StreamExt::boxed`
    --> /Users/kyle/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/stream/mod.rs:1331:23
     |
1329 |     fn boxed<'a>(self) -> BoxStream<'a, Self::Item>
     |        ----- required by a bound in this associated function
1330 |     where
1331 |         Self: Sized + Send + 'a,
     |                       ^^^^ required by this bound in `StreamExt::boxed`

error[E0277]: `*mut u8` cannot be sent between threads safely
    --> src/http/mod.rs:125:14
     |
125  |             .boxed();
     |              ^^^^^ `*mut u8` cannot be sent between threads safely
     |
     = help: within `futures::stream::Map<wasm_streams::readable::into_stream::IntoStream<'static>, [closure@reqwest::Response::bytes_stream::{closure#0}]>`, the trait `std::marker::Send` is not implemented for `*mut u8`
note: required because it appears within the type `PhantomData<*mut u8>`
    --> /Users/kyle/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/marker.rs:791:12
     |
791  | pub struct PhantomData<T: ?Sized>;
     |            ^^^^^^^^^^^
note: required because it appears within the type `JsValue`
    --> /Users/kyle/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasm-bindgen-0.2.87/src/lib.rs:92:12
     |
92   | pub struct JsValue {
     |            ^^^^^^^
note: required because it appears within the type `ReadableStreamGenericReader`
    --> /Users/kyle/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasm-streams-0.3.0/src/readable/sys.rs:143:14
     |
143  |     pub type ReadableStreamGenericReader;
     |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: required because it appears within the type `ReadableStreamDefaultReader`
    --> /Users/kyle/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasm-streams-0.3.0/src/readable/sys.rs:163:14
     |
163  |     pub type ReadableStreamDefaultReader;
     |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: required because it appears within the type `ReadableStreamDefaultReader<'static>`
    --> /Users/kyle/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasm-streams-0.3.0/src/readable/default_reader.rs:17:12
     |
17   | pub struct ReadableStreamDefaultReader<'stream> {
     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: required because it appears within the type `Option<ReadableStreamDefaultReader<'static>>`
    --> /Users/kyle/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/option.rs:563:10
     |
563  | pub enum Option<T> {
     |          ^^^^^^
note: required because it appears within the type `IntoStream<'static>`
    --> /Users/kyle/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wasm-streams-0.3.0/src/readable/into_stream.rs:22:12
     |
22   | pub struct IntoStream<'reader> {
     |            ^^^^^^^^^^
note: required because it appears within the type `Map<IntoStream<'static>, [closure@Response::bytes_stream::{closure#0}]>`
    --> /Users/kyle/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/stream/map.rs:15:16
     |
15   |     pub struct Map<St, F> {
     |                ^^^
     = note: required for `Unique<Map<IntoStream<'static>, ...>>` to implement `std::marker::Send`
     = note: the full type name has been written to '/Users/kyle/github/apache/arrow-rs/object_store/target/wasm32-unknown-unknown/debug/deps/object_store-87363e35aacc8eb7.long-type-262376943904190086.txt'
note: required because it appears within the type `Box<Map<IntoStream<'static>, [closure@Response::bytes_stream::{closure#0}]>>`
    --> /Users/kyle/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/alloc/src/boxed.rs:195:12
     |
195  | pub struct Box<
     |            ^^^
note: required because it appears within the type `Pin<Box<Map<IntoStream<'static>, [closure@Response::bytes_stream::{closure#0}]>>>`
    --> /Users/kyle/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/pin.rs:410:12
     |
410  | pub struct Pin<P> {
     |            ^^^
note: required because it appears within the type `impl Stream<Item = Result<Bytes, Error>>`
    --> /Users/kyle/.cargo/registry/src/index.crates.io-6f17d22bba15001f/reqwest-0.11.20/src/wasm/response.rs:137:34
     |
137  |     pub fn bytes_stream(self) -> impl futures_core::Stream<Item = crate::Result<Bytes>> {
     |                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: required because it appears within the type `IntoStream<impl Stream<Item = Result<Bytes, Error>>>`
    --> /Users/kyle/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/try_stream/into_stream.rs:12:16
     |
12   |     pub struct IntoStream<St> {
     |                ^^^^^^^^^^
note: required because it appears within the type `Map<IntoStream<impl Stream<Item = Result<Bytes, Error>>>, MapErrFn<[[email protected]:124:22]>>`
    --> /Users/kyle/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/stream/map.rs:15:16
     |
15   |     pub struct Map<St, F> {
     |                ^^^
note: required because it appears within the type `MapErr<impl Stream<Item = Result<Bytes, Error>>, [[email protected]:124:22]>`
    --> /Users/kyle/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/try_stream/mod.rs:63:5
     |
63   |     MapErr<St, F>(
     |     ^^^^^^
note: required by a bound in `futures::StreamExt::boxed`
    --> /Users/kyle/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/stream/stream/mod.rs:1331:23
     |
1329 |     fn boxed<'a>(self) -> BoxStream<'a, Self::Item>
     |        ----- required by a bound in this associated function
1330 |     where
1331 |         Self: Sized + Send + 'a,
     |                       ^^^^ required by this bound in `StreamExt::boxed`

error: future cannot be sent between threads safely
   --> src/http/mod.rs:134:65
    |
134 |       async fn head(&self, location: &Path) -> Result<ObjectMeta> {
    |  _________________________________________________________________^
135 | |         let status = self.client.list(Some(location), "0").await?;
136 | |         match status.response.len() {
137 | |             1 => {
...   |
152 | |         }
153 | |     }
    | |_____^ future created by async block is not `Send`
    |
    = help: within `[async block@src/http/mod.rs:134:65: 153:6]`, the trait `std::marker::Send` is not implemented for `*mut u8`
note: future is not `Send` as this value is used across an await
   --> src/http/client.rs:198:42
    |
198 |             Ok(result) => result.bytes().await.context(ReqwestSnafu)?,
    |                ------                    ^^^^^                      - `result` is later dropped here
    |                |                         |
    |                |                         await occurs here, with `result` maybe used later
    |                has type `reqwest::Response` which is not `Send`
    = note: required for the cast from `Pin<Box<[async block@src/http/mod.rs:134:65: 153:6]>>` to `Pin<Box<dyn Future<Output = Result<ObjectMeta, Error>> + Send>>`
    = note: the full name for the target type has been written to '/Users/kyle/github/apache/arrow-rs/object_store/target/wasm32-unknown-unknown/debug/deps/object_store-87363e35aacc8eb7.long-type-11917112363545009078.txt'

error: future cannot be sent between threads safely
   --> src/http/mod.rs:134:65
    |
134 |       async fn head(&self, location: &Path) -> Result<ObjectMeta> {
    |  _________________________________________________________________^
135 | |         let status = self.client.list(Some(location), "0").await?;
136 | |         match status.response.len() {
137 | |             1 => {
...   |
152 | |         }
153 | |     }
    | |_____^ future created by async block is not `Send`
    |
    = help: within `[async block@src/http/mod.rs:134:65: 153:6]`, the trait `std::marker::Send` is not implemented for `Rc<RefCell<wasm_bindgen_futures::Inner>>`
note: future is not `Send` as this value is used across an await
   --> /Users/kyle/.cargo/registry/src/index.crates.io-6f17d22bba15001f/reqwest-0.11.20/src/wasm/mod.rs:23:42
    |
23  |     let js_val = JsFuture::from(promise).await.map_err(crate::error::wasm)?;
    |                  ----------------------- ^^^^^                             - `JsFuture::from(promise)` is later dropped here
    |                  |                       |
    |                  |                       await occurs here, with `JsFuture::from(promise)` maybe used later
    |                  has type `wasm_bindgen_futures::JsFuture` which is not `Send`
    = note: required for the cast from `Pin<Box<[async block@src/http/mod.rs:134:65: 153:6]>>` to `Pin<Box<dyn Future<Output = Result<ObjectMeta, Error>> + Send>>`
    = note: the full name for the target type has been written to '/Users/kyle/github/apache/arrow-rs/object_store/target/wasm32-unknown-unknown/debug/deps/object_store-87363e35aacc8eb7.long-type-11917112363545009078.txt'

error: future cannot be sent between threads safely
   --> src/http/mod.rs:162:52
    |
162 |       ) -> Result<BoxStream<'_, Result<ObjectMeta>>> {
    |  ____________________________________________________^
163 | |         let prefix_len = prefix.map(|p| p.as_ref().len()).unwrap_or_default();
164 | |         let status = self.client.list(prefix, "infinity").await?;
165 | |         Ok(futures::stream::iter(
...   |
177 | |         .boxed())
178 | |     }
    | |_____^ future created by async block is not `Send`
    |
    = help: within `[async block@src/http/mod.rs:162:52: 178:6]`, the trait `std::marker::Send` is not implemented for `*mut u8`
note: future is not `Send` as this value is used across an await
   --> src/http/client.rs:198:42
    |
198 |             Ok(result) => result.bytes().await.context(ReqwestSnafu)?,
    |                ------                    ^^^^^                      - `result` is later dropped here
    |                |                         |
    |                |                         await occurs here, with `result` maybe used later
    |                has type `reqwest::Response` which is not `Send`
    = note: required for the cast from `Pin<Box<[async block@src/http/mod.rs:162:52: 178:6]>>` to `Pin<Box<dyn Future<Output = Result<Pin<Box<...>>, ...>> + Send>>`
    = note: the full name for the target type has been written to '/Users/kyle/github/apache/arrow-rs/object_store/target/wasm32-unknown-unknown/debug/deps/object_store-87363e35aacc8eb7.long-type-17290204358261428026.txt'

error: future cannot be sent between threads safely
   --> src/http/mod.rs:162:52
    |
162 |       ) -> Result<BoxStream<'_, Result<ObjectMeta>>> {
    |  ____________________________________________________^
163 | |         let prefix_len = prefix.map(|p| p.as_ref().len()).unwrap_or_default();
164 | |         let status = self.client.list(prefix, "infinity").await?;
165 | |         Ok(futures::stream::iter(
...   |
177 | |         .boxed())
178 | |     }
    | |_____^ future created by async block is not `Send`
    |
    = help: within `[async block@src/http/mod.rs:162:52: 178:6]`, the trait `std::marker::Send` is not implemented for `Rc<RefCell<wasm_bindgen_futures::Inner>>`
note: future is not `Send` as this value is used across an await
   --> /Users/kyle/.cargo/registry/src/index.crates.io-6f17d22bba15001f/reqwest-0.11.20/src/wasm/mod.rs:23:42
    |
23  |     let js_val = JsFuture::from(promise).await.map_err(crate::error::wasm)?;
    |                  ----------------------- ^^^^^                             - `JsFuture::from(promise)` is later dropped here
    |                  |                       |
    |                  |                       await occurs here, with `JsFuture::from(promise)` maybe used later
    |                  has type `wasm_bindgen_futures::JsFuture` which is not `Send`
    = note: required for the cast from `Pin<Box<[async block@src/http/mod.rs:162:52: 178:6]>>` to `Pin<Box<dyn Future<Output = Result<Pin<Box<...>>, ...>> + Send>>`
    = note: the full name for the target type has been written to '/Users/kyle/github/apache/arrow-rs/object_store/target/wasm32-unknown-unknown/debug/deps/object_store-87363e35aacc8eb7.long-type-17290204358261428026.txt'

error: future cannot be sent between threads safely
   --> src/http/mod.rs:180:86
    |
180 |       async fn list_with_delimiter(&self, prefix: Option<&Path>) -> Result<ListResult> {
    |  ______________________________________________________________________________________^
181 | |         let status = self.client.list(prefix, "1").await?;
182 | |         let prefix_len = prefix.map(|p| p.as_ref().len()).unwrap_or(0);
183 | |
...   |
209 | |         })
210 | |     }
    | |_____^ future created by async block is not `Send`
    |
    = help: within `[async block@src/http/mod.rs:180:86: 210:6]`, the trait `std::marker::Send` is not implemented for `*mut u8`
note: future is not `Send` as this value is used across an await
   --> src/http/client.rs:198:42
    |
198 |             Ok(result) => result.bytes().await.context(ReqwestSnafu)?,
    |                ------                    ^^^^^                      - `result` is later dropped here
    |                |                         |
    |                |                         await occurs here, with `result` maybe used later
    |                has type `reqwest::Response` which is not `Send`
    = note: required for the cast from `Pin<Box<[async block@src/http/mod.rs:180:86: 210:6]>>` to `Pin<Box<dyn Future<Output = Result<ListResult, Error>> + Send>>`
    = note: the full name for the target type has been written to '/Users/kyle/github/apache/arrow-rs/object_store/target/wasm32-unknown-unknown/debug/deps/object_store-87363e35aacc8eb7.long-type-7476912493252910664.txt'

error: future cannot be sent between threads safely
   --> src/http/mod.rs:180:86
    |
180 |       async fn list_with_delimiter(&self, prefix: Option<&Path>) -> Result<ListResult> {
    |  ______________________________________________________________________________________^
181 | |         let status = self.client.list(prefix, "1").await?;
182 | |         let prefix_len = prefix.map(|p| p.as_ref().len()).unwrap_or(0);
183 | |
...   |
209 | |         })
210 | |     }
    | |_____^ future created by async block is not `Send`
    |
    = help: within `[async block@src/http/mod.rs:180:86: 210:6]`, the trait `std::marker::Send` is not implemented for `Rc<RefCell<wasm_bindgen_futures::Inner>>`
note: future is not `Send` as this value is used across an await
   --> /Users/kyle/.cargo/registry/src/index.crates.io-6f17d22bba15001f/reqwest-0.11.20/src/wasm/mod.rs:23:42
    |
23  |     let js_val = JsFuture::from(promise).await.map_err(crate::error::wasm)?;
    |                  ----------------------- ^^^^^                             - `JsFuture::from(promise)` is later dropped here
    |                  |                       |
    |                  |                       await occurs here, with `JsFuture::from(promise)` maybe used later
    |                  has type `wasm_bindgen_futures::JsFuture` which is not `Send`
    = note: required for the cast from `Pin<Box<[async block@src/http/mod.rs:180:86: 210:6]>>` to `Pin<Box<dyn Future<Output = Result<ListResult, Error>> + Send>>`
    = note: the full name for the target type has been written to '/Users/kyle/github/apache/arrow-rs/object_store/target/wasm32-unknown-unknown/debug/deps/object_store-87363e35aacc8eb7.long-type-7476912493252910664.txt'

error: future cannot be sent between threads safely
   --> src/http/mod.rs:212:64
    |
212 |       async fn copy(&self, from: &Path, to: &Path) -> Result<()> {
    |  ________________________________________________________________^
213 | |         self.client.copy(from, to, true).await
214 | |     }
    | |_____^ future created by async block is not `Send`
    |
    = help: within `[async block@src/http/mod.rs:212:64: 214:6]`, the trait `std::marker::Send` is not implemented for `*mut u8`
note: future is not `Send` as this value is used across an await
   --> src/http/client.rs:284:60
    |
272 |             return match builder.send_retry(&self.retry_config).await {
    |                          -------------------------------------------- has type `std::result::Result<reqwest::Response, retry::Error>` which is not `Send`
...
284 |                         self.create_parent_directories(to).await?;
    |                                                            ^^^^^ await occurs here, with `builder.send_retry(&self.retry_config).await` maybe used later
...
289 |             };
    |              - `builder.send_retry(&self.retry_config).await` is later dropped here
    = note: required for the cast from `Pin<Box<[async block@src/http/mod.rs:212:64: 214:6]>>` to `Pin<Box<dyn Future<Output = Result<(), Error>> + Send>>`
    = note: the full name for the target type has been written to '/Users/kyle/github/apache/arrow-rs/object_store/target/wasm32-unknown-unknown/debug/deps/object_store-87363e35aacc8eb7.long-type-13573442131717126860.txt'

error: future cannot be sent between threads safely
   --> src/http/mod.rs:216:78
    |
216 |       async fn copy_if_not_exists(&self, from: &Path, to: &Path) -> Result<()> {
    |  ______________________________________________________________________________^
217 | |         self.client.copy(from, to, false).await
218 | |     }
    | |_____^ future created by async block is not `Send`
    |
    = help: within `[async block@src/http/mod.rs:216:78: 218:6]`, the trait `std::marker::Send` is not implemented for `*mut u8`
note: future is not `Send` as this value is used across an await
   --> src/http/client.rs:284:60
    |
272 |             return match builder.send_retry(&self.retry_config).await {
    |                          -------------------------------------------- has type `std::result::Result<reqwest::Response, retry::Error>` which is not `Send`
...
284 |                         self.create_parent_directories(to).await?;
    |                                                            ^^^^^ await occurs here, with `builder.send_retry(&self.retry_config).await` maybe used later
...
289 |             };
    |              - `builder.send_retry(&self.retry_config).await` is later dropped here
    = note: required for the cast from `Pin<Box<[async block@src/http/mod.rs:216:78: 218:6]>>` to `Pin<Box<dyn Future<Output = Result<(), Error>> + Send>>`
    = note: the full name for the target type has been written to '/Users/kyle/github/apache/arrow-rs/object_store/target/wasm32-unknown-unknown/debug/deps/object_store-87363e35aacc8eb7.long-type-16365264314614274671.txt'

error[E0599]: no method named `is_connect` found for reference `&reqwest::Error` in the current scope
  --> src/client/retry.rs:98:44
   |
98 |             (Some(source), None) if source.is_connect() => {
   |                                            ^^^^^^^^^^ method not found in `&Error`

error: future cannot be sent between threads safely
   --> src/client/retry.rs:162:9
    |
162 | /         async move {
163 | |             let mut retries = 0;
164 | |             let now = Instant::now();
165 | |
...   |
254 | |             }
255 | |         }
    | |_________^ future created by async block is not `Send`
    |
    = help: within `[async block@src/client/retry.rs:162:9: 255:10]`, the trait `std::marker::Send` is not implemented for `*mut u8`
note: future is not `Send` as this value is used across an await
   --> src/client/retry.rs:203:60
    |
169 |                     Ok(r) => match r.error_for_status_ref() {
    |                        - has type `reqwest::Response` which is not `Send`
...
203 |                                     true => match r.text().await {
    |                                                            ^^^^^ await occurs here, with `r` maybe used later
...
225 |                     },
    |                     - `r` is later dropped here
note: required by a bound in `futures::FutureExt::boxed`
   --> /Users/kyle/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/future/future/mod.rs:520:23
    |
518 |     fn boxed<'a>(self) -> BoxFuture<'a, Self::Output>
    |        ----- required by a bound in this associated function
519 |     where
520 |         Self: Sized + Send + 'a,
    |                       ^^^^ required by this bound in `FutureExt::boxed`

error: future cannot be sent between threads safely
   --> src/client/retry.rs:162:9
    |
162 | /         async move {
163 | |             let mut retries = 0;
164 | |             let now = Instant::now();
165 | |
...   |
254 | |             }
255 | |         }
    | |_________^ future created by async block is not `Send`
    |
    = help: within `reqwest::Response`, the trait `Sync` is not implemented for `*mut u8`
note: future is not `Send` as this value is used across an await
   --> src/client/retry.rs:203:60
    |
169 |                     Ok(r) => match r.error_for_status_ref() {
    |                                    - has type `&reqwest::Response` which is not `Send`
...
203 |                                     true => match r.text().await {
    |                                                            ^^^^^ await occurs here, with `r` maybe used later
...
225 |                     },
    |                     - `r` is later dropped here
help: consider moving this into a `let` binding to create a shorter lived borrow
   --> src/client/retry.rs:169:36
    |
169 |                     Ok(r) => match r.error_for_status_ref() {
    |                                    ^^^^^^^^^^^^^^^^^^^^^^^^
note: required by a bound in `futures::FutureExt::boxed`
   --> /Users/kyle/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/future/future/mod.rs:520:23
    |
518 |     fn boxed<'a>(self) -> BoxFuture<'a, Self::Output>
    |        ----- required by a bound in this associated function
519 |     where
520 |         Self: Sized + Send + 'a,
    |                       ^^^^ required by this bound in `FutureExt::boxed`

error: future cannot be sent between threads safely
   --> src/client/retry.rs:162:9
    |
162 | /         async move {
163 | |             let mut retries = 0;
164 | |             let now = Instant::now();
165 | |
...   |
254 | |             }
255 | |         }
    | |_________^ future created by async block is not `Send`
    |
    = help: within `[async block@src/client/retry.rs:162:9: 255:10]`, the trait `std::marker::Send` is not implemented for `Rc<RefCell<wasm_bindgen_futures::Inner>>`
note: future is not `Send` as this value is used across an await
   --> /Users/kyle/.cargo/registry/src/index.crates.io-6f17d22bba15001f/reqwest-0.11.20/src/wasm/mod.rs:23:42
    |
23  |     let js_val = JsFuture::from(promise).await.map_err(crate::error::wasm)?;
    |                  ----------------------- ^^^^^                             - `JsFuture::from(promise)` is later dropped here
    |                  |                       |
    |                  |                       await occurs here, with `JsFuture::from(promise)` maybe used later
    |                  has type `wasm_bindgen_futures::JsFuture` which is not `Send`
note: required by a bound in `futures::FutureExt::boxed`
   --> /Users/kyle/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.28/src/future/future/mod.rs:520:23
    |
518 |     fn boxed<'a>(self) -> BoxFuture<'a, Self::Output>
    |        ----- required by a bound in this associated function
519 |     where
520 |         Self: Sized + Send + 'a,
    |                       ^^^^ required by this bound in `FutureExt::boxed`

error[E0599]: no method named `user_agent` found for struct `ClientBuilder` in the current scope
   --> src/client/mod.rs:452:51
    |
452 |             Some(user_agent) => builder = builder.user_agent(user_agent.get()?),
    |                                                   ^^^^^^^^^^ method not found in `ClientBuilder`

error[E0599]: no method named `user_agent` found for struct `ClientBuilder` in the current scope
   --> src/client/mod.rs:453:39
    |
453 |             None => builder = builder.user_agent(DEFAULT_USER_AGENT),
    |                                       ^^^^^^^^^^ method not found in `ClientBuilder`

error[E0599]: no method named `add_root_certificate` found for struct `ClientBuilder` in the current scope
   --> src/client/mod.rs:468:35
    |
468 |                 builder = builder.add_root_certificate(certificate);
    |                                   ^^^^^^^^^^^^^^^^^^^^ method not found in `ClientBuilder`

error[E0599]: no method named `proxy` found for struct `ClientBuilder` in the current scope
   --> src/client/mod.rs:477:31
    |
477 |             builder = builder.proxy(proxy);
    |                               ^^^^^ method not found in `ClientBuilder`

error[E0599]: no method named `timeout` found for struct `ClientBuilder` in the current scope
   --> src/client/mod.rs:481:31
    |
481 |             builder = builder.timeout(timeout.get()?)
    |                               ^^^^^^^ method not found in `ClientBuilder`

error[E0599]: no method named `connect_timeout` found for struct `ClientBuilder` in the current scope
   --> src/client/mod.rs:485:31
    |
485 |             builder = builder.connect_timeout(timeout.get()?)
    |                               ^^^^^^^^^^^^^^^ method not found in `ClientBuilder`

error[E0599]: no method named `pool_idle_timeout` found for struct `ClientBuilder` in the current scope
   --> src/client/mod.rs:489:31
    |
489 |             builder = builder.pool_idle_timeout(timeout.get()?)
    |                               ^^^^^^^^^^^^^^^^^ method not found in `ClientBuilder`

error[E0599]: no method named `pool_max_idle_per_host` found for struct `ClientBuilder` in the current scope
   --> src/client/mod.rs:493:31
    |
493 |             builder = builder.pool_max_idle_per_host(max.get()?)
    |                               ^^^^^^^^^^^^^^^^^^^^^^ method not found in `ClientBuilder`

error[E0599]: no method named `http2_keep_alive_interval` found for struct `ClientBuilder` in the current scope
   --> src/client/mod.rs:497:31
    |
497 |             builder = builder.http2_keep_alive_interval(interval.get()?)
    |                               ^^^^^^^^^^^^^^^^^^^^^^^^^ method not found in `ClientBuilder`

error[E0599]: no method named `http2_keep_alive_timeout` found for struct `ClientBuilder` in the current scope
   --> src/client/mod.rs:501:31
    |
501 |             builder = builder.http2_keep_alive_timeout(interval.get()?)
    |                               ^^^^^^^^^^^^^^^^^^^^^^^^ method not found in `ClientBuilder`

error[E0599]: no method named `http2_keep_alive_while_idle` found for struct `ClientBuilder` in the current scope
   --> src/client/mod.rs:505:31
    |
505 |             builder = builder.http2_keep_alive_while_idle(true)
    |                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^ method not found in `ClientBuilder`

error[E0599]: no method named `http1_only` found for struct `ClientBuilder` in the current scope
   --> src/client/mod.rs:509:31
    |
509 |             builder = builder.http1_only()
    |                               ^^^^^^^^^^ method not found in `ClientBuilder`

error[E0599]: no method named `http2_prior_knowledge` found for struct `ClientBuilder` in the current scope
   --> src/client/mod.rs:513:31
    |
513 |             builder = builder.http2_prior_knowledge()
    |                               ^^^^^^^^^^^^^^^^^^^^^ method not found in `ClientBuilder`

error[E0599]: no method named `danger_accept_invalid_certs` found for struct `ClientBuilder` in the current scope
   --> src/client/mod.rs:517:31
    |
517 |             builder = builder.danger_accept_invalid_certs(true)
    |                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^ method not found in `ClientBuilder`

error[E0599]: no method named `https_only` found for struct `ClientBuilder` in the current scope
   --> src/client/mod.rs:521:14
    |
520 | /         builder
521 | |             .https_only(!self.allow_http.get()?)
    | |             -^^^^^^^^^^ method not found in `ClientBuilder`
    | |_____________|
    |

Some errors have detailed explanations: E0277, E0432, E0433, E0599.
For more information about an error, try `rustc --explain E0277`.

To Reproduce

git clone https://github.com/apache/arrow-rs
git checkout b66c57c4b4
cd object_store
cargo build -p object_store --target wasm32-unknown-unknown --features http

Expected behavior

Expected the crate to build with the http feature. The readme indicates that the aws, azure, and gcp features don't work in wasm, but I was hoping the http feature would.

Additional context

I have a wasm-bindgen library to read Parquet in WebAssembly and return an Arrow table to JavaScript: https://github.com/kylebarron/parquet-wasm. I have an existing implementation working using a RangedAsyncReader from https://github.com/DataEngineeringLabs/ranged-reader-rs but I was trying to test out the object store reader implementation to see if it would coalesce requests better. Right now I'm seeing a bunch of requests per row group, and especially from the browser coalescing requests is a high priority. See e.g. https://observablehq.com/d/f5723cea6661fb71, where I'm seeing 33 content requests for a parquet file with 7 row groups using the arrow-rs implementation. (For context, with the parquet2 implementation it appeared to make a separate request for each and every array in every row group)

The text was updated successfully, but these errors were encountered:

tustvold · 2023-09-05T08:53:32Z

This may just be documentation ommission, last time I tried the underlying HTTP client didn't support WASI (IIRC due to lack of tokio support). If this is no longer the case, we can probably support the full feature set, as the cloud features require little more than an http client as well

kylebarron · 2023-09-07T18:21:17Z

Do you need full tokio support? Some http libraries, e.g. reqwest support a wasm32 build, where they use the browser's JS calls as the base of the http requests (at least in the browser, the wasm sandbox isn't able to make its own HTTP calls)

tustvold · 2023-09-07T18:25:25Z

Quite probably not, but I've been unable to find the magic combination of feature flags, etc... to make this work. I would be more than happy to review PRs adding support for this, but do not personally intend to spend more time playing with this

tustvold · 2023-09-18T14:56:24Z

label_issue.py automatically added labels {'object-store'} from #4796

JanKaul · 2023-11-18T14:28:26Z

For time being, I've created an object_store implementation for s3 that works in wasm. (https://github.com/JanKaul/object_store_s3_wasm)

If you are intrested

tustvold · 2023-11-18T15:29:56Z

Interesting, I think the big thing that currently blocks supporting this first-party is that whilst reqwest does have a WASM32 client, it is effectively a completely independent API, with many methods we rely on missing, and more problematically no Send bounds on the returned futures.

I think it would be very cool if we could somehow decouple object_store from reqwest and allow use-provided HTTP clients, but I suspect this might be a fairly involved process

JanKaul · 2023-11-19T18:39:28Z

There already seems to be a discussion about this in the reqwest crate (seanmonstar/reqwest#1977). But this also seems like it's not happening any time soon.

kylebarron added the bug label Sep 5, 2023

kylebarron mentioned this issue Sep 5, 2023

Try using ParquetObjectReader for arrow1 async api kylebarron/parquet-wasm#312

Closed

tustvold added a commit to tustvold/arrow-rs that referenced this issue Sep 7, 2023

ObjectStore Wasm32 Fixes (apache#4775) (apache#4776)

becaa01

tustvold mentioned this issue Sep 7, 2023

ObjectStore Wasm32 Fixes (#4775) (#4776) #4796

Merged

alamb closed this as completed in #4796 Sep 14, 2023

alamb pushed a commit that referenced this issue Sep 14, 2023

ObjectStore Wasm32 Fixes (#4775) (#4776) (#4796)

229bf8b

tustvold added the object-store Object Store Interface label Sep 18, 2023

clbarnes mentioned this issue Nov 20, 2023

Rebase on object_store clbarnes/zarr3-rs#22

Open

KKould mentioned this issue Sep 2, 2024

Support WASM tonbo-io/tonbo#118

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

object-store fails to compile for `wasm32-unknown-unknown` with `http` feature #4776

object-store fails to compile for `wasm32-unknown-unknown` with `http` feature #4776

kylebarron commented Sep 5, 2023 •

edited

Loading

tustvold commented Sep 5, 2023

kylebarron commented Sep 7, 2023

tustvold commented Sep 7, 2023

tustvold commented Sep 18, 2023

JanKaul commented Nov 18, 2023

tustvold commented Nov 18, 2023

JanKaul commented Nov 19, 2023

object-store fails to compile for wasm32-unknown-unknown with http feature #4776

object-store fails to compile for wasm32-unknown-unknown with http feature #4776

Comments

kylebarron commented Sep 5, 2023 • edited Loading

tustvold commented Sep 5, 2023

kylebarron commented Sep 7, 2023

tustvold commented Sep 7, 2023

tustvold commented Sep 18, 2023

JanKaul commented Nov 18, 2023

tustvold commented Nov 18, 2023

JanKaul commented Nov 19, 2023

object-store fails to compile for `wasm32-unknown-unknown` with `http` feature #4776

object-store fails to compile for `wasm32-unknown-unknown` with `http` feature #4776

kylebarron commented Sep 5, 2023 •

edited

Loading