Asigning a field to a float

[MoAly98]

Hi,
I am trying to assign a new field in an awkward array to a float. To do that, I am simply using an equivalent call to

sum = ak.sum(array['y'])
array["x"] = sum

However, I am running into the following error from the with_field method:

  File "/afs/cern.ch/user/m/maly/.conda/envs/thbb_env/lib/python3.9/site-packages/awkward/highlevel.py", line 1068, in __setitem__
    array = ak.operations.structure.with_field(self.layout, what, where)
  File "/afs/cern.ch/user/m/maly/.conda/envs/thbb_env/lib/python3.9/site-packages/awkward/operations/structure.py", line 936, in with_field
    out = (ak.layout.RecordArray([what], [where], parameters=base.parameters),)
ValueError: content argument must be a Content subtype

I can get around the error by simply setting the field to be a numpy array with identical elements using np.full with a shape identical to array['y']. Is it a rule that I have to set the field to be an awkward array or a numpy array?

[jpivarski]

This has been working for me:

>>> array = ak.Array([{"y": 1}, {"y": 2}, {"y": 3}])
>>> sum = ak.sum(array['y'])
>>> array["x"] = sum
>>> array
<Array [{y: 1, x: 6}, ... x: 6}, {y: 3, x: 6}] type='3 * {"y": int64, "x": int64}'>

Considering that sum is already a defined symbol in Python, and it's a thing that can't be assigned into an ak.Array (<built-in function sum>), maybe your script is seeing the original sum, rather than the output of ak.sum(array['y'])?

I just did a quick check and see that the right-hand-side can be Python numbers (int, float) or NumPy numbers (np.int32, np.float32, etc.). That's not the problem.

[jpivarski]

Well, facetiously, it's possible if it broadcasts correctly!

In this case, it looks like you want the "x" that you assign to have the same length as "y", meaning the same number of [5, 6, 7] lists as "y" has numbers, since "x" is an array of lists and "y" is an array of numbers.

Now I'm a little worried that broadcasting might have been too strong a constraint—it might always try to make them have the same structure, the way it would have to if you were doing a numerical operation on the two arrays (such as +). In that case, my snide comment aside, it might actually be impossible to do that assignment, and we'd have to rethink how assignments are defined.

One thing that would always be possible is to use an Array constrictor,

ak.Array({"y": y, "x": x})

or ak.zip,

ak.zip({"y": y, "x": x}, depth_limit=1)

The depth_limit gives you control over how deeply it attempts to zip the arrays together, which is the control that might be lost if broadcasting is too strong a rule for assignment.

(In any case, don't worry about performance of making a new array vs attaching to an old array—they do the same thing internally, which is to share memory as much as possible.)

[MoAly98]

Hmmm i see. In my case, I have a very large array (~O(10^6) events) which already has fields that are arrays. I just want to add a field to it that is an array of some length such that each event gets an instance of that array. If I try to break down my original array, rebuild it as a dict then pass it to ak.Array I:

A) still get errors

dict of arrays in ak.Array constructor must have arrays of equal length

B) Code becomes very slow

My code:

new_data = {k: data[k] for k in data.fields }
new_data[new_key] = new_arr 
data = ak.Array(new_data)

using ak.zip solves the error, but does performance is still very bad due to the dict comprehension.

[agoose77]

How many fields does your array have?

[MoAly98]

This will vary. I'm writing a dumper which grabs any number of branches (specified by useR) out of root files, as well as create new ones. I'm currently testing with roughly 10 fields.

[jpivarski]

Just a quick demo that gluing arrays together with ak.zip (with depth_limit=1) is O(1) in time and memory. (Edit: ak.zip with depth_limit=1 should be equivalent to the ak.Array constructor, since the latter doesn't do any zipping.)

In [1]: import awkward as ak, numpy as np, psutil

In [2]: psutil.virtual_memory().percent
Out[2]: 24.9

In [3]: fields_1e6 = {f"f{i}": ak.Array(np.random.normal(0, 1, int(1e6))) for i
   ...: in range(10)}

In [4]: fields_1e8 = {f"f{i}": ak.Array(np.random.normal(0, 1, int(1e8))) for i
   ...: in range(10)}

In [5]: psutil.virtual_memory().percent
Out[5]: 73.6

In [6]: tmp = []

In [7]: %timeit tmp.append(ak.zip(fields_1e6, depth_limit=1))
353 µs ± 9.62 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

In [8]: len(tmp)
Out[8]: 8111

In [9]: psutil.virtual_memory().percent
Out[9]: 73.9

In [10]: %timeit tmp.append(ak.zip(fields_1e8, depth_limit=1))
354 µs ± 8.98 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

In [11]: len(tmp)
Out[11]: 16222

In [12]: psutil.virtual_memory().percent
Out[12]: 73.9

The fields_1e6 and especially fields_1e8 are using a lot of memory, collectively 50% of what I have on my computer (because psutil's memory reporting goes from 24.9 up to 73.6; that's real RAM, not swap). However, creating a new array by zipping them together is 350 µs regardless of whether I use the million-element arrays or the 100 million-element arrays. Also, they're persistently added to tmp and the memory usage doesn't appreciably increase. (The 0.3% increase could be web browser stuff in the background.)

If this had been assigning a field to an existing array or depth_limit != 1 (with arrays that have depth), then there would be some additional cost from checking to see if they broadcast, or creating a broadcasted array if it needs modification to do so, and those can be O(n).

The rule is that the output of an operation reuses as much of the input's buffers as possible, and the columnar layout of the arrays in memory allows for more opportunities to apply this technique. (It's called "structural sharing," a technique made popular by Rich Hickey in the design of Clojure, but it works even better for columnar data structures.)

---

Oh wait! Your 10 fields is just a small sample. You might have as many fields as a TTree, which can be up to thousands (some CMS and ATLAS files I've seen have 1000‒3000 TBranches).

Time and memory costs do scale with the number of fields, but O(f), not O(f × n) for f fields and arrays of length n. I'll do another run of that:

In [1]: import awkward as ak, numpy as np, psutil

In [2]: psutil.virtual_memory().percent
Out[2]: 25.2

In [3]: fields_10 = {f"f{i}": ak.Array(np.random.normal(0, 1, int(1e6))) for i i
   ...: n range(10)}

In [4]: fields_1000 = {f"f{i}": ak.Array(np.random.normal(0, 1, int(1e6))) for i
   ...:  in range(1000)}

In [5]: psutil.virtual_memory().percent
Out[5]: 73.7

In [6]: tmp = []

In [7]: %timeit tmp.append(ak.zip(fields_10, depth_limit=1))
354 µs ± 9.95 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

In [8]: len(tmp)
Out[8]: 8111

In [9]: psutil.virtual_memory().percent
Out[9]: 73.9

In [10]: %timeit tmp.append(ak.zip(fields_1000, depth_limit=1))
18.2 ms ± 130 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

In [11]: len(tmp)
Out[11]: 8922

In [12]: psutil.virtual_memory().percent
Out[12]: 75.6

In [13]: 18.2 / 0.354
Out[13]: 51.41242937853107

We can see the time cost: ak.zip on 1000 fields takes 18.2 ms, which is 50× slower than an array with 100× fewer fields (almost proportional). The memory cost should be a little bigger, too, because a std::map of 1000 items is larger than a std::map of 10 items. (In Awkward 2.0, these will be Python dicts, not C++ std::maps, and there would be more referencing and less copying.) However, %time created a lot fewer of the big ones, so they're hard to compare.

But the objects that are larger and slower are the "handles" to the big data. If your numeric data in arrays outweighs your administrative metadata (as it should be), then all of that dict manipulation for 1000 fields won't be a significant cost.

So—what's "slow" and how do you measure it? Sometimes, people use small arrays like [[1, 2, 3], [], [4, 5]] for performance tests and compare µs times to conclude that an operation is too slow, but this isn't the right metric if you're eventually going to be working on large-n datasets.