[Draft] flutter_rust_bridge 2.0: Allow Dart to *own* and manage Rust's opaque objects

[fzyzcjy]

Related discussions: #68 (thanks @gcxfd)

Why

Powerful!

Problems to solve

What data structure to pass around?

Attempt 1: Maybe Box<Arc<RwLock<T>>> as a raw pointer?

However, can we do that? When raw pointer is passed from dart to multiple rust functions concurrently, multiple rust functions will all rehydrate the same Arc object from the raw pointer, so have problems?

Attempt 2: Maybe Box<(AtomicUsize, RwLock<T>)> as a raw pointer? the AtomicUsize is like a self-managed Arc.

Attempt 3: Maybe Box<RwLock<T>> is enough? Since when it is used in a Rust function, the Rust function always immutably borrow the RwLock<T>, and it is Dart who indeed owns it. --> See "How to ensure that the Dart opaque object is still alive before" below. This method is WRONG.

How can we create a Dart object with a finalizer, and then pass that object from Rust to Dart?

use as_native_pointer of Dart_CObject https://github.com/dart-lang/sdk/blob/5f32f4d7b678dc08d6079664637ad061902c5b69/runtime/include/dart_native_api.h#L88

Interestingly, it was added only 4months ago! dart-lang/sdk@bc38783#diff-587868a9cf3a0dd424f04d4d4d30efef5f00661441e094b1e993d20d3847199d

In other words, in Dart 2.14 (Flutter 2.5), https://github.com/dart-lang/sdk/blob/2.14.0/runtime/include/dart_native_api.h does NOT have it.
In Dart 2.15 (Flutter 2.8 - released a few days ago!) https://github.com/dart-lang/sdk/blob/2.15.0/runtime/include/dart_native_api.h It is there now.

How to update the real "external size" of the Rust object to Dart, when using the Dart_CObject instead of DartHandle?

see: dart-lang/sdk#47901

[TODO]

How to know the real "external size" of the Rust object?

It is needed by Dart VM, otherwise vm may not know the memory pressure (I guess?).

For example, if the user's data structure is struct MyObject {a: i32, b: String, c: Box<Something>}, how can we know the actual heap size used? Notice String and Box...

Attempt 1: Maybe create a trait and enforce user to implement that trait. Say, pub trait Sizeable { fn memory_size(&self) -> usize }.

However, how to know that size has changed in Rust, and tell Dart the new size whenever it has changed (maybe using Dart_UpdateExternalSize/Dart_UpdateFinalizableExternalSize)?

Attempt 2: Not only have the trait Sizeable, but also require the user to manually call flutter_rust_bridge when its memory size changes.

Not that friendly...

Attempt 3: Can we automatically generate some code?

Attempt 4: Is there a library to do this?

https://crates.io/crates/deepsize

Related issue: Aeledfyr/deepsize#28

Problem: Can it accurately know things like ndarray? Aeledfyr/deepsize#29

Attempt 5: Can we just use Attempt 2 and tolerate that the users may not update the size frequently enough?

see dart-lang/sdk#47900

Will we face performance problems?

Look at how ObjC makes ref counting, sounds a bit similar.

May not be a problem, since we have to have interior mutability, and have to be thread safe.

How to ensure that the Dart opaque object is still alive when Rust is using that object? Especially, when in async

Serious problem!

So we really need Arc to have reference counting, instead of blindly let Dart to own the object. After doing so, we simply do not care when is Dart object deallocated.

Do we have concurrecy problems?

Sounds no, since we have RwLock.

What if, when Dart calls Rust, at the same time, Dart makes a GC and make that pointer invalid?

Code example:

// Dart
var opaque_pointer = ...;
my_ffi_function(opaque_pointer);

// Rust
fn my_ffi_function(opaque_pointer) {
  opaque_pointer.convert_raw_pointer_into_rust_object().increase_arc_count(); // (a)
}

When we are executing (a), what if Dart make a GC and the finalizer of opaque_pointer is called, then we are trying to do increase_arc_count on an invalid object? (Notice if the finalizer is called and the pointer has rc=1, then it is dropped immediately)

Attempt 1: Maybe force the opaque_pointer to be used after the ffi call is done. For example:

var opaque_pointer = ...;
await my_ffi_function(opaque_pointer);
pretend_we_use_it_to_avoid_gc(opaque_pointer); // <--- NOTE

[TODO] Seems to have seen some discussions about this before?

[raphaelrobert]

This is what I had in mind for #230, except this approach is more versatile and hopefully robust.
I have experimented with this myself by just passing an integer (of at least the size of usize) from Rust to Dart. It generally works well, but obviously you have to be careful regarding concurrency and GC.

Solving this one (and #67) would make this framework truly powerful! The Rust/Dart FFI in larger projects I have seen so far always involved having a server on the Rust side and a client on the Dart side. While that works, it makes for a lot of boilerplate code and general overhead.

I can't pretend I understand the internals of flutter_rust_bridge well enough to do a PR, but I'd offer to help with debugging for example.

[raphaelrobert]

On the subject of Dart-side GC: When dart-lang/sdk#47772 lands, it should be possible to call Rust when Dart decides to do GC. It looks to me like Arc would then work well. Having RwLock in addition to that might just work for concurrency.

[fzyzcjy]

@raphaelrobert Thank you!

[raphaelrobert]

I tried to do a PoC with Arc<RwLock<T>>, because it looks like Box is not needed since Arc already does heap allocations: https://gist.github.com/raphaelrobert/f7e517a7d4d02cd4b6d24cb270207791

I think this does all that's needed:

• threat safe
• allow for multiple instantiations when the pointer gets copied on the Dart side
• free memory when Dart disposes of the pointer (manually at first, later this could be automated with Dart finalizers)

Gist

Opaque PoC

Opaque PoC. GitHub Gist: instantly share code, notes, and snippets.

[fzyzcjy]

@raphaelrobert Great job - that PoC sounds wonderful!

free memory when Dart disposes of the pointer (manually at first, later this could be automated with Dart finalizers)

Automatically indeed ;) Look at this:

https://github.com/dart-lang/sdk/blob/5f32f4d7b678dc08d6079664637ad061902c5b69/runtime/include/dart_native_api.h#L84-L88

...
    struct {
      intptr_t ptr;
      intptr_t size;
      Dart_HandleFinalizer callback;
    } as_native_pointer;
...

We can create a Dart_CObject (which is the type of object that this lib return from Rust to Dart) of the type "native_pointer". Then we have a callback which will automatically called when Dart wants to finalize the opaque pointer.

Remark about Dart_CObject: For example, in Rust we can create Dart_CObject of type array and several Dart_CObjects inside that array, and post it to Dart (via allo_isolate crate - which is thin wrapper of dart api). Then for example, in Dart we can get something like [42, "hello", 1.23, [1,2,3]] etc

GitHub

sdk/dart_native_api.h at 5f32f4d7b678dc08d6079664637ad061902c5b69 · dart-lang/sdk

The Dart SDK, including the VM, dart2js, core libraries, and more. - sdk/dart_native_api.h at 5f32f4d7b678dc08d6079664637ad061902c5b69 · dart-lang/sdk

[raphaelrobert]

Yes, I saw that native finalizers were already available, but I wasn't sure if they could be used in this case. If they can, all the better!

[fzyzcjy]

@raphaelrobert

Yes, I saw that native finalizers were already available, but I wasn't sure if they could be used in this case. If they can, all the better!

I guess yes, at least in Flutter 2.8. They are added only in 2.8 and is not there for 2.5

[raphaelrobert]

@raphaelrobert

Yes, I saw that native finalizers were already available, but I wasn't sure if they could be used in this case. If they can, all the better!

I guess yes, at least in Flutter 2.8. They are added only in 2.8 and is not there for 2.5

If shekohex/allo-isolate#18 gets merged, we should be one step closer.

[raphaelrobert]

After testing the new NativePointer from the allo-isolate, I can say that it generally seems to work.

In more detail:

• Returning the NativePointer from Rust causes Dart to see this an int
• The int contains the raw value of the pointer (an isize in Rust)
• The int can therefore be used in subsequent calls to Rust, however it's really just a reference. On the Rust side you have to call mem::forget() before returning, otherwise Dart will segfault. This makes sense, since the ownership is now fully on the Dart side.
• In theory the callback gets called when Dart does GC. However, I did not manage to reliably trigger this. I ran the Dart code with dart run --observe to have access to the debugger where GC can be manually triggered. This had no effect whatsoever.
• I also tried to allocate several GB on the Rust side and reported the size back to Dart in order to create greater memory pressure. Dart counts the Rust heap in the RSS part of the memory, not its own heap (which actually makes sense).
• Dart's GC strategy is quite obscure, I expected a manual trigger to call the finalizer, especially when the int was completely out of scope. I even tried to do all of it in a dedicated isolate, to no avail.
• The only time when the callback was actually called, was when I inadvertently threw an exception after having received the int. In that instance The Rust callback was executed correctly, I inspected the memory before dropping the object and everything looked good.

To summarize, there are quite a few unknowns and things seem to be in motion on the Dart side. I'm wondering if it makes sense to investigate further at this point or whether it is better to wait for Dart finalizers to become available and look at it again then.

[fzyzcjy]

@raphaelrobert Great insights!

Interesting. I do not know it returns a int. Originally I thought it should return something like ffi.Pointer. How do you find that it returns a int?

The GC is quite strange. Have you tried to allocate millions of small objects that are not used (referenced) in Dart? By doing so maybe we can trigger a gc.

I'm wondering if it makes sense to investigate further at this point or whether it is better to wait for Dart finalizers to become available and look at it again then.

Good question. At first thought seems that dart finalizer is only porting the C finalizer to Dart; but since it involves lots of code, I guess Dart team is doing something more than that. So maybe Dart finalizer will be great, but I am also not sure...

Indeed, even if we have a Dart finalizer, we still have to give Dart a native pointer. Otherwise, we cannot tell Dart how big the external memory is.

[raphaelrobert]

Interesting. I do not know it returns a int. Originally I thought it should return something like ffi.Pointer. How do you find that it returns a int?

I implemented it and checked the runtimeType on the Dart side.

The GC is quite strange. Have you tried to allocate millions of small objects that are not used (referenced) in Dart? By doing so maybe we can trigger a gc.

Just allocating things on the Rust heap without passing them to Dart would not trigger it I think, because the Dart VM doesn't even know about them. It might be different if I did it on the Dart heap, but it's beside the point somehow.

[fzyzcjy]

Just allocating things on the Rust heap without passing them to Dart would not trigger it I think, because the Dart VM doesn't even know about them.

I mean allocate in Dart, not Rust. e.g. for(var i=0;i<100000;++i) var x = new Something();

[raphaelrobert]

I tried that and it worked as expected in the sense that GC kicked in, but only freed stuff on the Dart heap. The finalizers never got called.

I brought this up on the Dart SDK issue list btw: dart-lang/sdk#48023

[fzyzcjy]

@raphaelrobert Ok so I guess it is really a problem of Dart/Flutter's DartCObject's finalizers...

[fzyzcjy]

@raphaelrobert By the way, what about providing some sample code (e.g. a sample github repo), such that me and dart people can have a look (e.g. a bug in the code)

[raphaelrobert]

Sure: https://github.com/raphaelrobert/dart_rust_ffi_poc

GitHub

GitHub - raphaelrobert/dart_rust_ffi_poc

Contribute to raphaelrobert/dart_rust_ffi_poc development by creating an account on GitHub.

[fzyzcjy]

Forward an interesting remark:

I would strongly discourage anyone from using GC to manage non-Dart objects. If you want to manage native object lifetimes, have an explicit method like close or dispose to release native resources. GC might not ever run, or run too late.

dart-lang/language#1847 (comment)

by @dnfield

[fzyzcjy]

More insights by @dnfield: dart-lang/language#1847 (comment)

The VM can be aware of external object size, but GC pressure comes from new allocations. It's really not that hard to get into a scenario where you have an external object that takes up very close to a ceiling of memory you want to use, but not enough to trigger a GC. Then, new allocations happen quickly and kick off a GC, but you then run out of memory (or file handles, or some other native limited resource) before the GC can finish.

Flutter had this problem with images for example - we were relying on the GC to clean them up, which works pretty often but does not work so well when you try to load larger images in memory constrained environments. The more we tried to make the GC clean this up for us, the worse it got - the GC sometimes couldn't work fast enough (so new images could get allocated before the GC-able ones got cleaned up, leading to OOMs), and we also were artificially running the GC too often (Because it saw these huge objects it thought it was responsible for cleaning up). So now we explicitly and eagerly dispose images/graphics resources, and you can't get into that race anymore (and we don't need as many GCs, which are pretty resource intensive to run).

[raphaelrobert]

I tried to experiment with external typed data and and GC'ing works as expected: it occurs when allocating new objects, but there is no guarantee it will clean up everything.

With all this new information I think neither native pointers nor externally typed data should be the core of the solution. If we want to have opaque pointers (and I still think we should), we need to do that through more code generation. We probably need Dart classes that contain private raw pointers and an explicit destructor to free the memory on the native heap. When native finalizers have landed, we can still use those additionally to free up the native memory but we shouldn't rely on them for all the reasons mentioned above.

[Desdaemon]

I've made an informal RFC at #293, feel free to discuss it there as well.

[fzyzcjy]

By the way, this repo https://github.com/dart-native/dart_native is implementing bridge for Dart vs ObjC and used fully automatic memory management.

I have told him about the discourage from Dart team: dart-native/dart_native#88

[fzyzcjy]

(Sad) updates:

External size is just an input to GC scheduling heuristic.
However please don't use this type. It was introduced for some internal usage and has somewhat confusing semantics.

dart-lang/sdk#47900

Please don't use this type. It was introduced for some internal usage and has somewhat confusing semantics.

dart-lang/sdk#47901

In short: We should NOT use type Dart_CObject_kNativePointer.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[sagudev]

Any progress?

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new issue.

[fzyzcjy]

Related:

• Opaque type support #775

[fzyzcjy]

flutter_rust_bridge is really there!

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new issue.