Port `PendingRow` to `arrow-rs` #8617

emilk · 2025-01-08T15:26:37Z

Part of Migrate from re_arrow2 to arrow #3741
Requires More conversions to and from arrow-rs re_arrow2#15

NOTE: this will regress on memory use, because we lack .shrink_to_fit, which will arrive in

Update arrow, pyo3, and numpy #8618

github-actions · 2025-01-08T19:39:13Z

Web viewer built successfully. If applicable, you should also test it:

I have tested the web viewer

Result	Commit	Link	Manifest
✅	`21d76fd`	https://rerun.io/viewer/pr/8617	`+nightly` `+main`

^{Note: This comment is updated whenever you push a commit.}

emilk · 2025-01-08T19:46:05Z

crates/store/re_chunk/src/arrow_util.rs

+/// * The list only contains null entries, or empty arrays, or a mix of both.
+#[inline]
+pub fn is_list_array_semantically_empty(list_array: &ListArray) -> bool {
+    list_array.values().is_empty()


I took the freedom to simplify this function quite a bit from the original:

rerun/crates/store/re_chunk/src/util.rs

Lines 22 to 37 in 71215db

pub fn is_list_array_semantically_empty(list_array: &ArrowListArray<i32>) -> bool {

let is_physically_empty = || list_array.is_empty();

let is_all_nulls = || {

list_array

.validity()

.map_or(false, |bitmap| bitmap.unset_bits() == list_array.len())

};

let is_all_empties = || list_array.offsets().lengths().all(|len| len == 0);

let is_a_mix_of_nulls_and_empties =

|| list_array.iter().flatten().all(|array| array.is_empty());

is_physically_empty() || is_all_nulls() || is_all_empties() || is_a_mix_of_nulls_and_empties()

}

Was the old complexity necessary for some reason I don't see, or an oversight?

All I can say is that this logic was very important for the UX of the dataframe views and APIs.

Whether the two implementations are equivalent, I'm tempted to say yes but I might be missing something too.

emilk · 2025-01-08T19:47:00Z

crates/store/re_chunk/src/arrow_util.rs

@@ -0,0 +1,317 @@
+use arrow::{


This whole file is a port of https://github.com/rerun-io/rerun/blob/main/crates/store/re_chunk

emilk · 2025-01-08T19:59:34Z

crates/store/re_chunk/src/arrow_util.rs

+pub fn concat_arrays(arrays: &[&dyn Array]) -> arrow::error::Result<ArrayRef> {
+    #[allow(clippy::disallowed_methods)] // that's the whole point
+    arrow::compute::concat(arrays)
+    // TODO: call .shrink_to_fit on the result


Requires Update arrow, pyo3, and numpy #8618

…but maybe it's fine to regress on this temporarily

emilk · 2025-01-09T09:27:19Z

@rerun-bot full-check

github-actions · 2025-01-09T09:27:49Z

Started a full build: https://github.com/rerun-io/rerun/actions/runs/12687112287

teh-cmc

Gave it a very shallow look but it all seems fairly mechanical so there shouldn't too many surprises.

A full-check will definitely tell us if something went wrong in any case.

teh-cmc · 2025-01-09T09:30:22Z

crates/store/re_chunk/src/arrow_util.rs

+/// * The list only contains null entries, or empty arrays, or a mix of both.
+#[inline]
+pub fn is_list_array_semantically_empty(list_array: &ListArray) -> bool {
+    list_array.values().is_empty()


All I can say is that this logic was very important for the UX of the dataframe views and APIs.

Whether the two implementations are equivalent, I'm tempted to say yes but I might be missing something too.

emilk · 2025-01-09T09:39:52Z

@rerun-bot full-check

github-actions · 2025-01-09T09:40:25Z

Started a full build: https://github.com/rerun-io/rerun/actions/runs/12687313927

emilk · 2025-01-09T09:53:20Z

Deploying the Python docs failed the first run, but worked the second go

emilk added 5 commits January 8, 2025 11:12

Use Arrow2 prefix for arrow2 types

8882a81

Move arrow2-utils into module called arrow2_util

ef6654c

Create an arrow_util.rs

8d8638a

Port PendingRow to arrow-rs

288552d

Update tests

55df031

emilk force-pushed the emilk/pending-row branch from 3b09fb8 to 55df031 Compare January 8, 2025 19:38

emilk added 🏹 arrow concerning arrow do-not-merge Do not merge this PR 🚜 refactor Change the code, not the functionality exclude from changelog PRs with this won't show up in CHANGELOG.md labels Jan 8, 2025

emilk added 2 commits January 8, 2025 20:41

Revert renamings in arrow2_util

32817ba

insert_descriptor_arrow2

d98fe58

emilk commented Jan 8, 2025

View reviewed changes

emilk added 2 commits January 8, 2025 20:50

Remove insert_descriptor_arrow2 again

78f676f

Update list of disallowed methods

4be004c

emilk commented Jan 8, 2025

View reviewed changes

emilk marked this pull request as ready for review January 8, 2025 19:59

emilk changed the title ~~WIP: arrow2 shenanigans~~ Port PendingRow to arrow-rs Jan 8, 2025

teh-cmc self-requested a review January 9, 2025 09:06

Use main re_arrow2

d056aa0

teh-cmc approved these changes Jan 9, 2025

View reviewed changes

Update some tests

21d76fd

emilk removed the do-not-merge Do not merge this PR label Jan 9, 2025

emilk mentioned this pull request Jan 8, 2025

Migrate from re_arrow2 to arrow #3741

Open

14 tasks

emilk merged commit 3a50bc2 into main Jan 9, 2025
66 of 67 checks passed

emilk deleted the emilk/pending-row branch January 9, 2025 12:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Port `PendingRow` to `arrow-rs` #8617

Port `PendingRow` to `arrow-rs` #8617

emilk commented Jan 8, 2025 •

edited

Loading

github-actions bot commented Jan 8, 2025 •

edited

Loading

emilk Jan 8, 2025

teh-cmc Jan 9, 2025

emilk Jan 8, 2025

emilk Jan 8, 2025 •

edited

Loading

emilk commented Jan 9, 2025

github-actions bot commented Jan 9, 2025

teh-cmc left a comment

teh-cmc Jan 9, 2025

emilk commented Jan 9, 2025

github-actions bot commented Jan 9, 2025

emilk commented Jan 9, 2025 •

edited

Loading

	pub fn is_list_array_semantically_empty(list_array: &ArrowListArray<i32>) -> bool {
	let is_physically_empty = \|\| list_array.is_empty();

	let is_all_nulls = \|\| {
	list_array
	.validity()
	.map_or(false, \|bitmap\| bitmap.unset_bits() == list_array.len())
	};

	let is_all_empties = \|\| list_array.offsets().lengths().all(\|len\| len == 0);

	let is_a_mix_of_nulls_and_empties =
	\|\| list_array.iter().flatten().all(\|array\| array.is_empty());

	is_physically_empty() \|\| is_all_nulls() \|\| is_all_empties() \|\| is_a_mix_of_nulls_and_empties()
	}

Port PendingRow to arrow-rs #8617

Port PendingRow to arrow-rs #8617

Conversation

emilk commented Jan 8, 2025 • edited Loading

github-actions bot commented Jan 8, 2025 • edited Loading

emilk Jan 8, 2025

Choose a reason for hiding this comment

teh-cmc Jan 9, 2025

Choose a reason for hiding this comment

emilk Jan 8, 2025

Choose a reason for hiding this comment

emilk Jan 8, 2025 • edited Loading

Choose a reason for hiding this comment

emilk commented Jan 9, 2025

github-actions bot commented Jan 9, 2025

teh-cmc left a comment

Choose a reason for hiding this comment

teh-cmc Jan 9, 2025

Choose a reason for hiding this comment

emilk commented Jan 9, 2025

github-actions bot commented Jan 9, 2025

emilk commented Jan 9, 2025 • edited Loading

Port `PendingRow` to `arrow-rs` #8617

Port `PendingRow` to `arrow-rs` #8617

emilk commented Jan 8, 2025 •

edited

Loading

github-actions bot commented Jan 8, 2025 •

edited

Loading

emilk Jan 8, 2025 •

edited

Loading

emilk commented Jan 9, 2025 •

edited

Loading