High memory usage when calculating variance? #18824
Labels
bug
Something isn't working
needs triage
Awaiting prioritization by a maintainer
python
Related to Python Polars
Checks
Reproducible example
Log output
No log output?
Issue description
Potential culprit might be
squared
inpolars/crates/polars-core/src/chunked_array/ops/aggregate/var.rs
Lines 17 to 32 in 9384945
I'm not a stats/numerical analysis expert so I'm not sure whether the extra allocation is strictly necessary for numerical stability/other reasons. It feels like it shouldn't be necessary though?
Looks like NumPy has a similar issue? numpy/numpy#13199 (var, std memory consumption)
Expected behavior
Expected memory usage to not jump when calculating the variance
Installed versions
The text was updated successfully, but these errors were encountered: