Analysis of Emission Pipeline Top 20% and Bottom 80%

[TeachMeTW]

Introduces two distinct performance analysis methods to evaluate function-level metrics within our emission dataset. The objective is to identify which functions significantly impact performance and which do not, enabling targeted optimizations and improvements.

Analysis Types

1. Individual Entry Categorization

• Purpose: Categorizes each individual data.reading entry into Top 20% or Bottom 80% based on the 80th percentile within each data.name group.
• Use Case: Identifies specific high-impact executions of functions, allowing for pinpointing problematic instances.

Features

• Exclusions: Specific functions are excluded as they are parent of smaller functions and provide no insights:
  • TRIP_SEGMENTATION/segment_into_trips
  • TRIP_SEGMENTATION/segment_into_trips_dist/loop
• Sorting: Both Top 20% and Bottom 80% categories are sorted in descending order of data.reading for easy identification of high-impact entries.

---

2. Aggregated Entry Categorization

• Purpose: Aggregates data.reading metrics (both sum and mean) for each data.name and categorizes the aggregated values into Top 20% and Bottom 80% based on their respective 80th percentiles.
• Use Case: Determines which functions are consistently resource-intensive on average or cumulatively, providing a broader view of performance impact.

Features

• Aggregation Types:
  • Sum Aggregation: Total data.reading per function.
  • Mean Aggregation: Average data.reading per function.
• Sorting: Both Top 20% and Bottom 80% categories are sorted in descending order of aggregated data.reading.

---

bottom80_function_level_individual_sorted.csv
bottom80_function_level_mean_sorted.csv
bottom80_function_level_sum_sorted.csv
top20_function_level_individual_sorted.csv
top20_function_level_mean_sorted.csv
top20_function_level_sum_sorted.csv

[TeachMeTW]

From both mean and sum, the same functions show up in the top 20% and bottom 80% albeit a different order. I will go ahead and prune them from the pipeline.

[shankari]

@TeachMeTW please put the results in graphs or inline tables so that we don't have to keep downloading files to see the results.

Where are the scripts that you used to generate this split? Please link to them here.
Without transparency in the scripts, you have no proof of the results

[shankari]

I also don't see any updates in here after including the time filter. The top 20% contains only the dist filter results.
And the files were uploaded "two weeks ago" which was the time that the dist filter analysis was done.

[TeachMeTW]

December Update

• Instrumented create_places_and_trips and handle_trip_ended functions.
• Collected and analyzed updated performance metrics for both top-performing (Top 20%) and lesser-performing (Bottom 80%) operations.

Previous Data (based on ccebikes prod)

Top 20%

data.name,average_reading
TRIP_SEGMENTATION/create_places_and_trips,6.11492777755484
TRIP_SEGMENTATION/segment_into_trips_time/has_trip_ended,5.203754482825234
TRIP_SEGMENTATION/segment_into_trips_dist/has_trip_ended,0.33176563871859305
TRIP_SEGMENTATION/get_data_df,0.13479221123112303

Bottom 80%

data.name,average_reading
TRIP_SEGMENTATION/segment_into_trips_dist/get_filtered_points_df,0.1277177627025931
TRIP_SEGMENTATION/segment_into_trips_dist/get_transition_df,0.12171991904162699
TRIP_SEGMENTATION/segment_into_trips_time/get_filtered_points_pre_ts_diff_df,0.06102312830432008
TRIP_SEGMENTATION/segment_into_trips_time/get_transition_df,0.04896719856575752
TRIP_SEGMENTATION/segment_into_trips_dist/post_loop,0.009637999072394989
TRIP_SEGMENTATION/segment_into_trips_time/calculations_per_iteration,0.009030038257105083
TRIP_SEGMENTATION/get_time_range_for_segmentation,0.008812385345143932
TRIP_SEGMENTATION/segment_into_trips_dist/check_transitions_post_loop,0.0054819285690497896
TRIP_SEGMENTATION/handle_out_of_order_points,0.004389608453097595
TRIP_SEGMENTATION/segment_into_trips_time/post_loop,0.001586464757565409
TRIP_SEGMENTATION/segment_into_trips_time/filter_bogus_points,0.000983266447049876
TRIP_SEGMENTATION/segment_into_trips_dist/handle_trip_end,0.000659627289595929
TRIP_SEGMENTATION/segment_into_trips_dist/mark_valid,0.0003925704017833427
TRIP_SEGMENTATION/segment_into_trips_dist/get_last_trip_end_point,0.0003441714427687905
TRIP_SEGMENTATION/segment_into_trips_dist/continue_just_ended,0.00030632289442217953
TRIP_SEGMENTATION/get_filters_in_df,0.00028325290716931125
TRIP_SEGMENTATION/get_time_series,1.9164622159294e-05
TRIP_SEGMENTATION/create_dist_filter,4.974840094848555e-06
TRIP_SEGMENTATION/create_time_filter,4.684489195038672e-06
TRIP_SEGMENTATION/segment_into_trips_dist/set_new_trip_start_point,1.5871754537026088e-06

New Instrumented (Based on local open_access)

Top 20%

data.name,average_reading
TRIP_SEGMENTATION/get_combined_segmentation_points,289.41001524999956
TRIP_SEGMENTATION/segment_into_trips_dist/get_filtered_points_df,49.97330445059812
TRIP_SEGMENTATION/create_places_and_trips/process_segments,49.43672608266319
TRIP_SEGMENTATION/get_data_df,9.957696244051705

Bottom 80%

data.name,average_reading
TRIP_SEGMENTATION/segment_into_trips_time/has_trip_ended,0.008631514248452451
TRIP_SEGMENTATION/segment_into_trips_time/has_trip_ended/motion_and_gap_checks,0.0018074113038215288
TRIP_SEGMENTATION/create_places_and_trips/initialization,0.0017730383579070694
TRIP_SEGMENTATION/create_places_and_trips/insert_last_place,0.0007796396894767679
TRIP_SEGMENTATION/segment_into_trips_time/has_trip_ended/final_distance_checks,0.00021890854319204695
TRIP_SEGMENTATION/create_places_and_trips/log_segmentation_points,8.484674730508489e-05
TRIP_SEGMENTATION/segment_into_trips_time/has_trip_ended/initial_checks,2.015703723259139e-05

Local Staging

Top 20%

data.name,average_reading
TRIP_SEGMENTATION/get_combined_segmentation_points,244.59239484766667
TRIP_SEGMENTATION/segment_into_trips_dist/get_filtered_points_df,1.3070380361337925
TRIP_SEGMENTATION/create_places_and_trips/process_segments,0.956691232625004
TRIP_SEGMENTATION/segment_into_trips_dist/get_transition_df,0.6485740562857245
TRIP_SEGMENTATION/get_data_df,0.5813163733474915

Bottom 80%

data.name,average_reading
TRIP_SEGMENTATION/segment_into_trips_time/has_trip_ended,0.04396951262024141
TRIP_SEGMENTATION/segment_into_trips_time/has_trip_ended/motion_and_gap_checks,0.029915469854730245
TRIP_SEGMENTATION/handle_out_of_order_points,0.02109187234964338
TRIP_SEGMENTATION/segment_into_trips_dist/has_trip_ended,0.01407521516264799
TRIP_SEGMENTATION/segment_into_trips_time/has_trip_ended/tracking_restart_check,0.010045099816818958
TRIP_SEGMENTATION/segment_into_trips_dist/post_loop,0.008481668595777592
TRIP_SEGMENTATION/get_time_range_for_segmentation,0.007696437147387769
TRIP_SEGMENTATION/segment_into_trips_dist/check_transitions_post_loop,0.00346295049530454
TRIP_SEGMENTATION/create_places_and_trips/process_segments/handle_untracked,0.0031489393157869906
TRIP_SEGMENTATION/get_filters_in_df,0.0019108789019810501
TRIP_SEGMENTATION/create_places_and_trips/initialization,0.0011224947499962241
TRIP_SEGMENTATION/segment_into_trips_dist/handle_trip_end,0.00076395126471166
TRIP_SEGMENTATION/create_places_and_trips/insert_last_place,0.0006669287500017754
TRIP_SEGMENTATION/segment_into_trips_dist/continue_just_ended,0.0005316707959574592
TRIP_SEGMENTATION/segment_into_trips_dist/mark_valid,0.00048122658699867316
TRIP_SEGMENTATION/segment_into_trips_dist/get_last_trip_end_point,0.00035555957177610593
TRIP_SEGMENTATION/segment_into_trips_time/has_trip_ended/final_distance_checks,0.00025557633905966227
TRIP_SEGMENTATION/create_places_and_trips/log_segmentation_points,6.484904166289181e-05
TRIP_SEGMENTATION/segment_into_trips_time/has_trip_ended/initial_checks,2.116130941705208e-05
TRIP_SEGMENTATION/get_time_series,1.9063383660977706e-05
TRIP_SEGMENTATION/create_time_filter,6.11470386502333e-06
TRIP_SEGMENTATION/create_dist_filter,3.921490133507177e-06
TRIP_SEGMENTATION/segment_into_trips_dist/set_new_trip_start_point,1.380765752401203e-06

It seems that get_combined_segmentation_points is a big pain point -- however, to note this wasn't present on production runs possibly because of this block:

        # Common case - let's make it easy
        with ect.Timer() as t_segment_trips:
            segmentation_points = filter_methods[filters_in_df[0]].segment_into_trips(ts, time_query)
        esds.store_pipeline_time(user_id, ecwp.PipelineStages.TRIP_SEGMENTATION.name + "/segment_into_trips", time.time(), t_segment_trips.elapsed)
    else:
        with ect.Timer() as t_get_combined_segmentation:
            segmentation_points = get_combined_segmentation_points(ts, loc_df, time_query,
                                                                   filters_in_df,
                                                                   filter_methods)
        esds.store_pipeline_time(user_id, ecwp.PipelineStages.TRIP_SEGMENTATION.name + "/get_combined_segmentation_points", time.time(), t_get_combined_segmentation.elapsed)

[TeachMeTW]

Addressing Performance Bottlenecks and Instrumentation Insights

1. Loading Big Datasets and Rerunning the Pipeline

I have identified create_places_and_trips and has_trip_ended as the main bottlenecks based on the data from the server datasets. These areas were instrumented locally, and the pipeline was cleared and rerun using large datasets, such as those from staging and open-access environments.

2. Verifying the Same Areas as Bottlenecks

The bottlenecks observed locally are not the same as those identified in the production and staging environments. This discrepancy raises questions about whether the current instrumentation fully reflects server-side performance issues.

3. Relative Performance of Bottlenecks Locally

While the areas under scrutiny should theoretically remain proportional, the performance discrepancies suggest that local instrumentation may not provide an accurate proportional representation of bottlenecks seen in production or the server.

4. Using Logs for Granular Refinement

The new instrumentation does not seem to reflect server-side statistics accurately, leaving some uncertainty about next steps. Thoughts on this matter would be appreciated @shankari @JGreenlee

[TeachMeTW]

For example, in the local runs, there is this new stat: TRIP_SEGMENTATION/get_combined_segmentation_points,289.41001524999956 that was NOT present in the server runs. Should I instrument this? It is only triggered when if len(filters_in_df) != 1

Otherwise, the other areas of interet is things like TRIP_SEGMENTATION/segment_into_trips_dist/get_filtered_points_df,0.1277177627025931 where it is proportional and available on all datasets that may be improved

[JGreenlee]

Interesting findings. I took some time to digest this and I think we can conclude that right now, you should attempt to optimize these 2 blocks which were in the top 20% both locally and on prod:

• TRIP_SEGMENTATION/create_places_and_trips
  • Your new local instrumentation revealed that the slowest sub-block was TRIP_SEGMENTATION/create_places_and_trips/process_segments, so focus there
• TRIP_SEGMENTATION/get_data_df which was in the top 20%

As for TRIP_SEGMENTATION/segment_into_trips_time/has_trip_ended and TRIP_SEGMENTATION/segment_into_trips_dist/has_trip_ended not being as slow locally as they are on prod, I don't know all the specific reasons for the disparity – except that the nrel-hosted DB on prod (AWS documentDB) is generally slower than a local mongoDB.
shankari has said before that there is a way to throttle MongoDB so it runs slower locally. Maybe if we try that, we will see more proportional readings?

Nonetheless, I think the first thing to do is optimize get_data_df and create_places_and_trips/process_segments