BlockSource Requiring Send trait

[TonyGiorgio]

In order to implement BlockSource, we need to implement AsyncBlockSourceResult for its return type, which requires Send.

pub type AsyncBlockSourceResult<'a, T> = Pin<Box<dyn Future<Output = BlockSourceResult<T>> + 'a + Send>>;

For other projects I've used, this hasn't been a problem. However I'm working on a wasm project now and we get some errors around the Send trait when trying to use BDK:

Example code:

    fn get_block<'a>(
        &'a self,
        header_hash: &'a BlockHash,
    ) -> AsyncBlockSourceResult<'a, BlockData> {
        Box::pin(async move {
            let res_opt = self.blockchain.get_block_raw(header_hash).await.unwrap();

            match res_opt {
                Some(res) => Ok(FullBlock(res)),
                None => Err(BlockSourceError::transient(ChainAccessFailed)),
            }
        })
    }

error: future cannot be sent between threads safely
   --> src/wallet.rs:171:9                                                           
    |                                                                                
171 | /         Box::pin(async {                                                     
172 | |             // May want to use BlockSourceError::transient instead depending on the bdk::Error variant
173 | |             let height = self                                                
174 | |                 .blockchain                                                  
...   |                                                                              
185 | |             Ok(tuple)                                                        
186 | |         })                                                                   
    | |__________^ future created by async block is not `Send`
    |                                                                                
    = help: within `impl futures::Future<Output = Result<(BlockHash, std::option::Option<u32>), BlockSourceError>>`, the trait `std::marker::Send` is not implemented for `Rc<RefCell<wasm_bindgen_futures::Inner>>`
note: future is not `Send` as this value is used across an await
   --> /home/dev/.cargo/registry/src/github.com-1ecc6299db9ec823/reqwest-0.11.12/src/wasm/mod.rs:22:41
    |                                                                                
22  |     let js_val = JsFuture::from(promise).await.map_err(crate::error::wasm)?;
    |                  -----------------------^^^^^^ await occurs here, with `JsFuture::from(promise)` maybe used later
    |                  |                                                             
    |                  has type `JsFuture` which is not `Send`
note: `JsFuture::from(promise)` is later dropped here
   --> /home/dev/.cargo/registry/src/github.com-1ecc6299db9ec823/reqwest-0.11.12/src/wasm/mod.rs:22:76
    |                                                                                
22  |     let js_val = JsFuture::from(promise).await.map_err(crate::error::wasm)?;
    |                                                                            ^
    = note: required for the cast from `impl futures::Future<Output = Result<(BlockHash, std::option::Option<u32>), BlockSourceError>>` to the object type `dyn futures::Future<Output = Result<(BlockHash, std::option::Option<u32>), BlockSourceError>> + std::marker::Send`

I think the important one to pay attention to here is:

has type `JsFuture` which is not `Send`

We still have a lot to learn about wasm-bindgen and rust in general, but it looks like it implements !Send but not Send? https://rustwasm.github.io/wasm-bindgen/api/wasm_bindgen_futures/struct.JsFuture.html

---

I guess I wanted to open up discussion on some options here we were considering.

1. Just use the Confirm trait instead - we noticed this comment: Add get raw block endpoint bitcoindevkit/rust-esplora-client#25 (review)
2. Figure out how to get JsFuture to be Send compatible
3. Figure out if LDK needs to have Send here or not.

[jkczyz]

Are there any steps to reproduce this? cargo check gives me:

The following warnings were emitted during compilation:

warning: error: unable to create target: 'No available targets are compatible with triple "wasm32-unknown-unknown"'
warning: 1 error generated.

error: failed to run custom build command for `secp256k1-sys v0.6.1`

Caused by:
  process didn't exit successfully: `/Users/jkczyz/src/mutiny-web-poc/node-manager/target/debug/build/secp256k1-sys-b6b7b610c4561ccb/build-script-build` (exit status: 1)
  --- stdout
  TARGET = Some("wasm32-unknown-unknown")
  OPT_LEVEL = Some("0")
  HOST = Some("x86_64-apple-darwin")
  cargo:rerun-if-env-changed=CC_wasm32-unknown-unknown
  CC_wasm32-unknown-unknown = None
  cargo:rerun-if-env-changed=CC_wasm32_unknown_unknown
  CC_wasm32_unknown_unknown = None
  cargo:rerun-if-env-changed=TARGET_CC
  TARGET_CC = None
  cargo:rerun-if-env-changed=CC
  CC = None
  cargo:rerun-if-env-changed=CFLAGS_wasm32-unknown-unknown
  CFLAGS_wasm32-unknown-unknown = None
  cargo:rerun-if-env-changed=CFLAGS_wasm32_unknown_unknown
  CFLAGS_wasm32_unknown_unknown = None
  cargo:rerun-if-env-changed=TARGET_CFLAGS
  TARGET_CFLAGS = None
  cargo:rerun-if-env-changed=CFLAGS
  CFLAGS = None
  cargo:rerun-if-env-changed=CRATE_CC_NO_DEFAULTS
  CRATE_CC_NO_DEFAULTS = None
  DEBUG = Some("true")
  cargo:rerun-if-env-changed=CC_wasm32-unknown-unknown
  CC_wasm32-unknown-unknown = None
  cargo:rerun-if-env-changed=CC_wasm32_unknown_unknown
  CC_wasm32_unknown_unknown = None
  cargo:rerun-if-env-changed=TARGET_CC
  TARGET_CC = None
  cargo:rerun-if-env-changed=CC
  CC = None
  cargo:rerun-if-env-changed=CFLAGS_wasm32-unknown-unknown
  CFLAGS_wasm32-unknown-unknown = None
  cargo:rerun-if-env-changed=CFLAGS_wasm32_unknown_unknown
  CFLAGS_wasm32_unknown_unknown = None
  cargo:rerun-if-env-changed=TARGET_CFLAGS
  TARGET_CFLAGS = None
  cargo:rerun-if-env-changed=CFLAGS
  CFLAGS = None
  cargo:rerun-if-env-changed=CRATE_CC_NO_DEFAULTS
  CRATE_CC_NO_DEFAULTS = None
  running: "clang" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "--target=wasm32-unknown-unknown" "-I" "depend/secp256k1/" "-I" "depend/secp256k1/include" "-I" "depend/secp256k1/src" "-I" "wasm/wasm-sysroot" "-Wall" "-Wextra" "-DSECP256K1_API=" "-DENABLE_MODULE_ECDH=1" "-DENABLE_MODULE_SCHNORRSIG=1" "-DENABLE_MODULE_EXTRAKEYS=1" "-DUSE_NUM_NONE=1" "-DUSE_FIELD_INV_BUILTIN=1" "-DUSE_SCALAR_INV_BUILTIN=1" "-DECMULT_GEN_PREC_BITS=4" "-DECMULT_WINDOW_SIZE=15" "-DUSE_EXTERNAL_DEFAULT_CALLBACKS=1" "-DENABLE_MODULE_RECOVERY=1" "-o" "/Users/jkczyz/src/mutiny-web-poc/node-manager/target/wasm32-unknown-unknown/debug/build/secp256k1-sys-238e57c829dfa334/out/wasm/wasm.o" "-c" "wasm/wasm.c"
  cargo:warning=error: unable to create target: 'No available targets are compatible with triple "wasm32-unknown-unknown"'
  cargo:warning=1 error generated.
  exit status: 1

  --- stderr


  error occurred: Command "clang" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "--target=wasm32-unknown-unknown" "-I" "depend/secp256k1/" "-I" "depend/secp256k1/include" "-I" "depend/secp256k1/src" "-I" "wasm/wasm-sysroot" "-Wall" "-Wextra" "-DSECP256K1_API=" "-DENABLE_MODULE_ECDH=1" "-DENABLE_MODULE_SCHNORRSIG=1" "-DENABLE_MODULE_EXTRAKEYS=1" "-DUSE_NUM_NONE=1" "-DUSE_FIELD_INV_BUILTIN=1" "-DUSE_SCALAR_INV_BUILTIN=1" "-DECMULT_GEN_PREC_BITS=4" "-DECMULT_WINDOW_SIZE=15" "-DUSE_EXTERNAL_DEFAULT_CALLBACKS=1" "-DENABLE_MODULE_RECOVERY=1" "-o" "/Users/jkczyz/src/mutiny-web-poc/node-manager/target/wasm32-unknown-unknown/debug/build/secp256k1-sys-238e57c829dfa334/out/wasm/wasm.o" "-c" "wasm/wasm.c" with args "clang" did not execute successfully (status code exit status: 1).

[TonyGiorgio]

I think that's because of llvm issues with secp256k1: rust-bitcoin/rust-secp256k1#283

I think @futurepaul solved it on his mac by doing brew install llvm. There's also this just command that he uses afterwards: https://github.com/BitcoinDevShop/mutiny-web-poc/blob/5a1034b3e3f8747490e60deeea1caa42ba3c9066/justfile#L11

[jkczyz]

Getting somewhere but still not there:

$ AR=/opt/homebrew/opt/llvm/bin/llvm-ar CC=/opt/homebrew/opt/llvm/bin/clang wasm-pack test --headless --chrome ./node-manager
Error: Error during execution of `cargo metadata`:     Updating crates.io index
    Updating git repository `https://github.com/benthecarman/bdk`
    Updating git repository `https://github.com/benthecarman/rust-esplora-client`
error: failed to select a version for `gloo-utils`.
    ... required by package `gloo-net v0.2.4`
    ... which satisfies dependency `gloo-net = "^0.2.4"` of package `node-manager v0.1.0 (/Users/jkczyz/src/mutiny-web-poc/node-manager)`
versions that meet the requirements `^0.1` are: 0.1.4, 0.1.3, 0.1.2, 0.1.1, 0.1.0

the package `gloo-net` depends on `gloo-utils`, with features: `serde` but `gloo-utils` does not have these features.


failed to select a version for `gloo-utils` which could resolve this conflict

[TheBlueMatt]

Grrr, that's more than a little annoying. This really shouldn't matter at all - there's no threading in wasm, so Send and !Send are basically equivalent - you can't send something between threads if there's no threads to begin with!

Really we're being limited by lack of impl Future in traits - we should be able to drop the dyn in the AsyncBlockSourceResult type which would then let the compiler figure out whether we need/are Send or not based on the concrete type rather than a bound.

There's no easy way around this upstream until then (maybe eventually, now that GATs are stable?) - we'd really have to split the BlockSource trait into BlockSourceThreaded and BlockSourceNotSend, and then duplicate everything downstream from there.

You should be able to work around it downstream, though. If you wrap the JsFuture in something which provides Send (eg just struct Ugh(JsFuture); unsafe impl Send for Ugh {}) it should work just fine.