-
Notifications
You must be signed in to change notification settings - Fork 31
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix minor bug in event_model's rechunk_event_pages #297
Conversation
Ahh, I spoke too soon! It turns out that rechunk_event_pages still fails if you increase the chunk size. So, if you input a list of EventPages and want to increase the chunk size to decrease the number of pages, the constructor for EventPages also needs to have a non-empty "filled" key. I need to fix this as well, so don't merge yet. |
Ok, now I can chunk up or down. I wonder if some tests should be implemented for this... |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for the fix, @cjtitus.
Please fix the formatting which became broken due to the fix.
If you could add a simple test to demonstrate the bug and its fix, it would be great!
Ok, formatting fixed, and I added parametrization to the I have checked that this test fails on the master branch (and passes for this PR) |
Issues were addressed in follow-up commits
rechunk_event_pages
has a bug if you try to re-chunk an EventPage that has an empty "filled" key, because the dictionary comprehension to create the new "filled" dictionaries uses keys from "data". This means that you can't rechunk the EventPages that are emitted by the RunEngine.Description
I have just changed one word -- instead of
"filled": {key: page["filled"][key][start:stop] for key in page["data"].keys()},
I have written
"filled": {key: page["filled"][key][start:stop] for key in page["filled"].keys()},
which correctly leaves filled empty, if it was empty to begin with.
Motivation and Context
It is necessary to fix this bug in order to re-chunk EventPages that are currently too large to go into Kafka after long fly-scans.
How Has This Been Tested?
I have tested this by manually calling rechunk_event_pages on some generated EventPages, and also by running the built-in tests.