merging parameters during concatenation

[grst]

Hi all,

here's another issue that I encountered in the context of AwkwardArrayViews in scverse/anndata#647, and I'm not sure how to best solve it. I guess one option would be to wait for #1391, and implement views with that.

Let's say, I want to concatenate a view with an empty array:

class AwkwardArrayView:
    pass
ak.behavior["AwkwardArrayView"] = AwkwardArrayView

view = ak.Array([{'a': 2}, {'a': 3}])
view = ak.with_parameter(view, "__array__", "AwkwardArrayView")

array = ak.Array([None])
res1 = ak.concatenate([view, array])
res2 = ak.concatenate([array, view])

For res1, I get

[{a: 2},
 {a: 3},
 None]
------------------------------------------------
type: 3 * option[{
    a: int64
}, parameters={"__array__": "AwkwardArrayView"}]

which I can slice like this:

>>> res1[:, 'a']
[2,
 3,
 None]
----------------
type: 3 * ?int64

However, for res2 (reversed concatenation), I get

>>> res2
[None,
 {a: 2},
 {a: 3}]
----------------------------------------------------
type: 3 * union[
    ?unknown,
    struct[{
        a: int64
    }, parameters={"__array__": "AwkwardArrayView"}]
]

>>> res2[:, 'a']
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
Input In [49], in <cell line: 1>()
----> 1 res2[:, 'a']

Stacktrace

File ~/anaconda3/envs/anndata_dev/lib/python3.9/site-packages/awkward/highlevel.py:956, in Array.__getitem__(self, where)
    527 """
    528 Args:
    529     where (many types supported; see below): Index of positions to
   (...)
    953 have the same dimension as the array being indexed.
    954 """
    955 with ak._errors.SlicingErrorContext(self, where):
--> 956     out = self._layout[where]
    957     if isinstance(out, ak.contents.NumpyArray):
    958         array_param = out.parameter("__array__")

File ~/anaconda3/envs/anndata_dev/lib/python3.9/site-packages/awkward/contents/content.py:509, in Content.__getitem__(self, where)
    508 def __getitem__(self, where):
--> 509     return self._getitem(where)

File ~/anaconda3/envs/anndata_dev/lib/python3.9/site-packages/awkward/contents/content.py:546, in Content._getitem(self, where)
    537 nextwhere = ak._slicing.prepare_advanced_indexing(items)
    539 next = ak.contents.RegularArray(
    540     self,
    541     self.length if self._backend.nplike.known_shape else 1,
    542     1,
    543     parameters=None,
    544 )
--> 546 out = next._getitem_next(nextwhere[0], nextwhere[1:], None)
    548 if out.length == 0:
    549     return out._getitem_nothing()

File ~/anaconda3/envs/anndata_dev/lib/python3.9/site-packages/awkward/contents/regulararray.py:416, in RegularArray._getitem_next(self, head, tail, advanced)
    412 nextcontent = self._content._carry(nextcarry, True)
    414 if advanced is None or advanced.length == 0:
    415     return RegularArray(
--> 416         nextcontent._getitem_next(nexthead, nexttail, advanced),
    417         nextsize,
    418         self._length,
    419         parameters=self._parameters,
    420     )
    421 else:
    422     nextadvanced = ak.index.Index64.empty(
    423         self._length * nextsize, self._backend.index_nplike
    424     )

File ~/anaconda3/envs/anndata_dev/lib/python3.9/site-packages/awkward/contents/unionarray.py:769, in UnionArray._getitem_next(self, head, tail, advanced)
    761     return UnionArray.simplified(
    762         self._tags,
    763         outindex,
    764         outcontents,
    765         parameters=self._parameters,
    766     )
    768 elif isinstance(head, str):
--> 769     return self._getitem_next_field(head, tail, advanced)
    771 elif isinstance(head, list):
    772     return self._getitem_next_fields(head, tail, advanced)

File ~/anaconda3/envs/anndata_dev/lib/python3.9/site-packages/awkward/contents/content.py:301, in Content._getitem_next_field(self, head, tail, advanced)
    299 def _getitem_next_field(self, head, tail, advanced: ak.index.Index | None):
    300     nexthead, nexttail = ak._slicing.headtail(tail)
--> 301     return self._getitem_field(head)._getitem_next(nexthead, nexttail, advanced)

File ~/anaconda3/envs/anndata_dev/lib/python3.9/site-packages/awkward/contents/unionarray.py:491, in UnionArray._getitem_field(self, where, only_fields)
    487 def _getitem_field(self, where, only_fields=()):
    488     return UnionArray.simplified(
    489         self._tags,
    490         self._index,
--> 491         [x._getitem_field(where, only_fields) for x in self._contents],
    492         parameters=None,
    493     )

File ~/anaconda3/envs/anndata_dev/lib/python3.9/site-packages/awkward/contents/unionarray.py:491, in <listcomp>(.0)
    487 def _getitem_field(self, where, only_fields=()):
    488     return UnionArray.simplified(
    489         self._tags,
    490         self._index,
--> 491         [x._getitem_field(where, only_fields) for x in self._contents],
    492         parameters=None,
    493     )

File ~/anaconda3/envs/anndata_dev/lib/python3.9/site-packages/awkward/contents/indexedoptionarray.py:253, in IndexedOptionArray._getitem_field(self, where, only_fields)
    250 def _getitem_field(self, where, only_fields=()):
    251     return IndexedOptionArray.simplified(
    252         self._index,
--> 253         self._content._getitem_field(where, only_fields),
    254         parameters=None,
    255     )

File ~/anaconda3/envs/anndata_dev/lib/python3.9/site-packages/awkward/contents/emptyarray.py:107, in EmptyArray._getitem_field(self, where, only_fields)
    106 def _getitem_field(self, where, only_fields=()):
--> 107     raise ak._errors.index_error(self, where, "not an array of records")

IndexError: while attempting to slice

    <Array [None, {a: 2}, {a: 3}] type='3 * union[?unknown, struct[{a: int6...'>

with

    (:, 'a')

at inner EmptyArray of length 0, using sub-slice 'a'.

Error details: not an array of records.

This is somewhat related to @ivirshup's observation in #2182 (comment) that concatenation may result in different types depending on the order of arguments.

If I get rid of the

view = ak.with_parameter(view, "__array__", "AwkwardArrayView")

line, the types are consistent.

[agoose77]

This is all strongly related to the bug in #2173 — EmptyArray is not entirely behaving like an identity in contexts that we might want it to. For context, this is what res1.type, res2.type prints in my branch #2179:

>>> print(res1.type, res2.type)
3 * ?{a: int64} 3 * ?{a: int64}

I am currently working on this, and will keep you updated :)

[jpivarski]

I was about to turn this into an issue. (I'll still do that, and you can mark it as resolved by #2179.)

Here's a somewhat more minimal reproducer for the issue (both would make good test cases):

>>> ak.concatenate((ak.with_parameter([1], "k", "v"), [None])).layout
<IndexedOptionArray len='2'>
    <parameter name='k'>'v'</parameter>
    <index><Index dtype='int64' len='2'>[ 0 -1]</Index></index>
    <content><NumpyArray dtype='int64' len='1'>[1]</NumpyArray></content>
</IndexedOptionArray>

but

>>> ak.concatenate(([None], ak.with_parameter([1], "k", "v"))).layout
<IndexedOptionArray len='2'>
    <index><Index dtype='int64' len='2'>[-1  0]</Index></index>
    <content><NumpyArray dtype='int64' len='1'>
        <parameter name='k'>'v'</parameter>
        [1]
    </NumpyArray></content>
</IndexedOptionArray>

The internal merge operation (which ak.concatenate invokes) should have a commutative property: x.mergemany([y, z]) should return the same type for any ordering of x, y, and z. (Of course the values will be in a different order.) It's difficult to ensure that because x is given a privileged place as self—ensuring it means comparing all of the mergemany implementations with each other.

As @grst demonstrated, it is definitely related to parameters:

>>> ak.concatenate(([1], [None])).layout
<IndexedOptionArray len='2'>
    <index><Index dtype='int64' len='2'>[ 0 -1]</Index></index>
    <content><NumpyArray dtype='int64' len='1'>[1]</NumpyArray></content>
</IndexedOptionArray>

and

>>> ak.concatenate(([None], [1])).layout
<IndexedOptionArray len='2'>
    <index><Index dtype='int64' len='2'>[-1  0]</Index></index>
    <content><NumpyArray dtype='int64' len='1'>[1]</NumpyArray></content>
</IndexedOptionArray>

It could be in the mergemany implementations, which @agoose77 is working on right now, but it's also possible that it's in the UnionArray.simplified implementation. That constructor needs to pick one of the contents to be the one that gets wrapped with option-type, and that choice breaks the symmetry among contents. (In examples like these, the UnionArray.simplified constructs a non-UnionArray, but I'm pretty sure that the mergemany code uses UnionArray.simplified to produce its final result.)