Added #121

[TeachMeTW]

Callback Error Updating store-trips.data - TypeError: 'float' object is not subscriptable

Description

We are encountering a TypeError while updating store-trips.data. The error traceback indicates that there is an issue with the function applied to the ble_sensed_summary column in the DataFrame. The error occurs because a float object is being treated as if it were a dictionary.

Traceback

Callback error updating store-trips.data Traceback (most recent call last): File "/usr/src/app/app_sidebar_collapsible.py", line 318, in update_store_trips df, user_input_cols = query_confirmed_trips(start_date, end_date, timezone) File "/usr/src/app/utils/db_utils.py", line 101, in query_confirmed_trips df["data.primary_ble_sensed_mode"] = df.ble_sensed_summary.apply(get_max_mode_from_summary) File "/root/miniconda-23.5.2/envs/emission/lib/python3.9/site-packages/pandas/core/series.py", line 4771, in apply return SeriesApply(self, func, convert_dtype, args, kwargs).apply() File "/root/miniconda-23.5.2/envs/emission/lib/python3.9/site-packages/pandas/core/apply.py", line 1123, in apply return self.apply_standard() File "/root/miniconda-23.5.2/envs/emission/lib/python3.9/site-packages/pandas/core/apply.py", line 1174, in apply_standard mapped = lib.map_infer( File "pandas/_libs/lib.pyx", line 2924, in pandas._libs.lib.map_infer File "/usr/src/app/utils/db_utils.py", line 97, in <lambda> get_max_mode_from_summary = lambda md: max(md["distance"], key=md["distance"].get) if len(md["distance"]) > 0 else "INVALID" TypeError: 'float' object is not subscriptable

Issue

The TypeError occurs because the lambda function get_max_mode_from_summary expects md to be a dictionary with a "distance" key that maps to another dictionary. However, it appears that some entries in ble_sensed_summary are float values, which are not subscriptable and thus lead to this error.

Expected Behavior

The ble_sensed_summary column should be processed correctly, and the get_max_mode_from_summary function should handle various data types properly without raising an error.

WAS NOT FOUND IN TESTING FOR 149

One user (me) had a 504 timeout error while @shankari discovered it was due to this issue.

Fixes #145 (comment)

[shankari]

@TeachMeTW can you highlight how this wasn't found in the testing for #149

[shankari]

One user (me) had a 504 timeout error while @shankari discovered it was due to this issue.

@TeachMeTW I would like to understand why you did not encounter this error in local testing

[TeachMeTW]

One user (me) had an error while @shankari discovered it was due to this issue.

@TeachMeTW I would like to understand why you did not encounter this error in local testing

In my local testing, I'm not entirely sure why the error didn't occur. Here's what I did:

• Function Invocation: I invoked the relevant function that might be related to the issue.
• Datasets Used: I loaded two different datasets: ca_e-bike and open_access.
• Testing Steps:
  • Navigated to the Trips tab and changed the date.
  • Changed the date both in the Data page and the Home page separately to ensure comprehensive testing.

Throughout these steps, no errors were encountered. It's possible that the issue is specific to the stage environment or certain data conditions that aren't present in my local setup.

Next Steps to Investigate:

1. Data Discrepancies:

   • Stage vs. Local Data: Verify if the stage datasets have any differences or edge cases not present locally.
   • Data Integrity: Check for any corrupted or unexpected data formats in the stage datasets.

2. Reproduce with Staging Data:

   • If possible, use a subset of the stage data in my local environment to see if the error can be replicated.

I'm open to any additional suggestions or insights that could help identify why the error isn't manifesting in my local tests but is affecting others. I'll make sure to ask @JGreenlee as to why I cannot reproduce the error

[TeachMeTW]

Maybe it's because I haven't reset my data? I've been using the same loaded open access and ca e-bike since the very beginning? They may have been modified -- I can try re intaking them and test that

[JGreenlee]

We had previously talked about this bug in #121 and e-mission/em-public-dashboard#141.
It is due to missing values in some datasets. It only affects certain programs.

My guess is that it affects Stage but it does not affect ca_e-bike or open_access. Thus, the local testing performed was not adequate to reproduce this issue.
I recommend performing local testing with a dump of Stage

[JGreenlee]

The commit in this PR appears to have been generated by copy-pasting code from #121. It would have been preferable to use git cherry-pick. However, I see that the indentation level has changed with the addition of Timer blocks, so maybe you tried git cherry-pick and it did not work.

Regardless, I manually compared this PR to #121 and they are the same except for the indentation level. Approved.

[TeachMeTW]

The commit in this PR appears to have been generated by copy-pasting code from #121. It would have been preferable to use git cherry-pick. However, I see that the indentation level has changed with the addition of Timer blocks, so maybe you tried git cherry-pick and it did not work.

Regardless, I manually compared this PR to #121 and they are the same except for the indentation level. Approved.

Yes I did try to cherry pick but there were merge conflicts in regards to the whole function

[TeachMeTW]

@JGreenlee @shankari I reproduced with openpath-stage-snapshot-oct-6 and was able to reproduce the error:

Seems Jack was right:

We had previously talked about this bug in #121 and e-mission/em-public-dashboard#141. It is due to missing values in some datasets. It only affects certain programs.

My guess is that it affects Stage but it does not affect ca_e-bike or open_access. Thus, the local testing performed was not adequate to reproduce this issue. I recommend performing local testing with a dump of Stage

It is dataset dependent