Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DRAFT: Weighted Average #833

Draft
wants to merge 23 commits into
base: main
Choose a base branch
from
Draft

DRAFT: Weighted Average #833

wants to merge 23 commits into from

Conversation

philipc2
Copy link
Member

@philipc2 philipc2 commented Jul 2, 2024 •
edited
Loading

Closes #826

Overview

Expected Usage

import uxarray as ux

grid_path = "/path/to/grid.nc"
data_path = "/path/to/data.nc"

uxds = ux.open_dataset(grid_path, data_path)

# this is how you use this function
some_output = uxds.some_function()

# this is another way to use this function
other_output = uxds.some_function(some_param = True)

PR Checklist

General

  • An issue is linked created and linked
  • Add appropriate labels
  • Filled out Overview and Expected Usage (if applicable) sections

Testing

  • Adequate tests are created if there is new functionality
  • Tests cover all possible logical paths in your function
  • Tests are not too basic (such as simply calling a function and nothing else)

Documentation

  • Docstrings have been added to all new functions
  • Docstrings have updated with any function changes
  • Internal functions have a preceding underscore (_) and have been added to docs/internal_api/index.rst
  • User functions have been added to docs/user_api/index.rst

Examples

  • Any new notebook examples added to docs/examples/ folder
  • Clear the output of all cells before committing
  • New notebook files added to docs/examples.rst toctree
  • New notebook files added to new entry in docs/gallery.yml with appropriate thumbnail photo in docs/_static/thumbnails/

@review-notebook-app Review Notebook App
Copy link

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

@philipc2
Copy link
Member Author

philipc2 commented Jul 2, 2024

@rytam2

I've set up the boilerplate for the weighted mean functionality. This should be a good place to get started. We can run over this during today's meeting.

@philipc2
Copy link
Member Author

philipc2 commented Jul 5, 2024

@rytam2

We have fixed the issue with the quad-hexagon grid. I've added it back to the test case.

@philipc2 philipc2 added the run-benchmark Run ASV benchmark workflow label Jul 17, 2024
@github-actions GitHub Actions
Copy link

github-actions bot commented Jul 17, 2024 •
edited
Loading

ASV Benchmarking

Benchmark Comparison Results

Benchmarks that have improved:

Change Before [b11d011] After [a0be1c1] Ratio Benchmark (Parameter)
- 445M 375M 0.84 face_bounds.FaceBounds.peakmem_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/quad-hexagon/grid.nc'))
- 467M 373M 0.80 mpas_ocean.Integrate.peakmem_integrate('480km')
failed 412±7μs n/a mpas_ocean.WeightedMean.time_weighted_mean_face_centered('120km')
failed 341±6μs n/a mpas_ocean.WeightedMean.time_weighted_mean_face_centered('480km')

Benchmarks that have stayed the same:

Change Before [b11d011] After [a0be1c1] Ratio Benchmark (Parameter)
375M 376M 1.00 face_bounds.FaceBounds.peakmem_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/mpas/QU/oQU480.231010.nc'))
375M 375M 1.00 face_bounds.FaceBounds.peakmem_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/scrip/outCSne8/outCSne8.nc'))
400M 379M 0.95 face_bounds.FaceBounds.peakmem_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/geoflow-small/grid.nc'))
1.59±0.02s 1.58±0.01s 1.00 face_bounds.FaceBounds.time_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/mpas/QU/oQU480.231010.nc'))
224±0.9ms 223±4ms 0.99 face_bounds.FaceBounds.time_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/scrip/outCSne8/outCSne8.nc'))
2.04±0.02s 2.01±0.02s 0.99 face_bounds.FaceBounds.time_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/geoflow-small/grid.nc'))
7.90±0.3ms 8.07±0.2ms 1.02 face_bounds.FaceBounds.time_face_bounds(PosixPath('/home/runner/work/uxarray/uxarray/test/meshfiles/ugrid/quad-hexagon/grid.nc'))
3.02±0.03s 3.08±0.03s 1.02 import.Imports.timeraw_import_uxarray
674±20ms 669±7ms 0.99 mpas_ocean.ConnectivityConstruction.time_face_face_connectivity('120km')
41.9±0.6ms 42.1±0.5ms 1.01 mpas_ocean.ConnectivityConstruction.time_face_face_connectivity('480km')
1.83±0.03ms 1.82±0.03ms 0.99 mpas_ocean.ConnectivityConstruction.time_n_nodes_per_face('120km')
538±10μs 557±20μs 1.03 mpas_ocean.ConnectivityConstruction.time_n_nodes_per_face('480km')
1.12±0μs 1.06±0μs 0.95 mpas_ocean.ConstructTreeStructures.time_ball_tree('120km')
280±1ns 270±2ns 0.96 mpas_ocean.ConstructTreeStructures.time_ball_tree('480km')
770±4ns 759±6ns 0.99 mpas_ocean.ConstructTreeStructures.time_kd_tree('120km')
270±1ns 257±2ns 0.95 mpas_ocean.ConstructTreeStructures.time_kd_tree('480km')
432M 432M 1.00 mpas_ocean.GeoDataFrame.peakmem_to_geodataframe('120km', False)
407M 407M 1.00 mpas_ocean.GeoDataFrame.peakmem_to_geodataframe('120km', True)
379M 379M 1.00 mpas_ocean.GeoDataFrame.peakmem_to_geodataframe('480km', False)
393M 377M 0.96 mpas_ocean.GeoDataFrame.peakmem_to_geodataframe('480km', True)
1.02±0.01s 1.03±0.01s 1.01 mpas_ocean.GeoDataFrame.time_to_geodataframe('120km', False)
53.2±0.4ms 52.7±0.4ms 0.99 mpas_ocean.GeoDataFrame.time_to_geodataframe('120km', True)
78.0±0.3ms 79.3±1ms 1.02 mpas_ocean.GeoDataFrame.time_to_geodataframe('480km', False)
5.50±0.2ms 5.51±0.08ms 1.00 mpas_ocean.GeoDataFrame.time_to_geodataframe('480km', True)
319M 321M 1.01 mpas_ocean.Gradient.peakmem_gradient('120km')
296M 296M 1.00 mpas_ocean.Gradient.peakmem_gradient('480km')
2.79±0.02ms 2.79±0.06ms 1.00 mpas_ocean.Gradient.time_gradient('120km')
308±5μs 320±6μs 1.04 mpas_ocean.Gradient.time_gradient('480km')
389M 389M 1.00 mpas_ocean.Integrate.peakmem_integrate('120km')
182±5ms 177±1ms 0.97 mpas_ocean.Integrate.time_integrate('120km')
12.0±0.04ms 12.1±0.05ms 1.00 mpas_ocean.Integrate.time_integrate('480km')
342±7ms 347±4ms 1.01 mpas_ocean.MatplotlibConversion.time_dataarray_to_polycollection('120km', 'exclude')
348±4ms 348±2ms 1.00 mpas_ocean.MatplotlibConversion.time_dataarray_to_polycollection('120km', 'include')
343±3ms 344±4ms 1.00 mpas_ocean.MatplotlibConversion.time_dataarray_to_polycollection('120km', 'split')
22.8±0.6ms 22.7±0.1ms 0.99 mpas_ocean.MatplotlibConversion.time_dataarray_to_polycollection('480km', 'exclude')
22.8±0.4ms 23.0±0.2ms 1.01 mpas_ocean.MatplotlibConversion.time_dataarray_to_polycollection('480km', 'include')
22.6±0.3ms 23.0±0.2ms 1.01 mpas_ocean.MatplotlibConversion.time_dataarray_to_polycollection('480km', 'split')
56.0±0.1ms 56.5±0.5ms 1.01 mpas_ocean.RemapDownsample.time_inverse_distance_weighted_remapping
45.7±0.2ms 45.9±0.2ms 1.01 mpas_ocean.RemapDownsample.time_nearest_neighbor_remapping
360±0.8ms 361±1ms 1.00 mpas_ocean.RemapUpsample.time_inverse_distance_weighted_remapping
266±2ms 264±0.2ms 0.99 mpas_ocean.RemapUpsample.time_nearest_neighbor_remapping
294M 294M 1.00 quad_hexagon.QuadHexagon.peakmem_open_dataset
291M 291M 1.00 quad_hexagon.QuadHexagon.peakmem_open_grid
5.58±0.2ms 6.24±0.5ms ~1.12 quad_hexagon.QuadHexagon.time_open_grid

Benchmarks that have got worse:

Change Before [b11d011] After [a0be1c1] Ratio Benchmark (Parameter)
+ 6.78±0.4ms 7.93±1ms 1.17 quad_hexagon.QuadHexagon.time_open_dataset

@rytam2 rytam2 mentioned this pull request Jul 26, 2024
Merged
14 tasks
rytam2 and others added 2 commits July 26, 2024 17:43
@rytam2
Committing weighted mean modifications from rtam/weighted mean to uxa…
0032870 
…rray/weighted-mean (#866)

* updated mean function with weighted arg

* updated weighted-mean functionality in dataarray.py

* edited weights to dask array

---------

Co-authored-by: Rachel Yuen Sum Tam <rtam@dec2454.hsn.de.hpc.ucar.edu>
Co-authored-by: Rachel Yuen Sum Tam <rtam@crhtc38.hpc.ucar.edu>
@philipc2
Merge branch 'main' into weighted-mean
8a2ecb3
@philipc2 philipc2 linked an issue Aug 12, 2024 that may be closed by this pull request
Weighted Mean Functionality #826
Open
philipc2
philipc2 commented Sep 13, 2024
View reviewed changes
uxarray/core/dataarray.py Outdated Show resolved Hide resolved
uxarray/core/dataarray.py Outdated Show resolved Hide resolved
uxarray/core/dataarray.py Outdated Show resolved Hide resolved
@philipc2 philipc2 removed the run-benchmark Run ASV benchmark workflow label Sep 15, 2024
@philipc2 philipc2 added the run-benchmark Run ASV benchmark workflow label Oct 10, 2024
philipc2
philipc2 commented Oct 22, 2024
View reviewed changes
test/test_weighted_mean.py
nt.assert_equal(result, expected_weighted_mean)


def test_csne30_equal_area():
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rytam2

Can you write a test case for this using Dask?

  • Face Areas & Data is a dask array

philipc2
philipc2 commented Oct 22, 2024
View reviewed changes
uxarray/core/dataarray.py
weighted_mean = (self * weights).sum(axis=-1) / total_weight

# create a UxDataArray and return it
return UxDataArray(weighted_mean, uxgrid=self.uxgrid)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to preserve other parameters:

  • Coordiantes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@rytam2 rytam2

Labels
run-benchmark Run ASV benchmark workflow
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Weighted Mean Functionality
2 participants
@philipc2 @rytam2