Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Show infinitely looping files in observability dashboard #155

Merged

Conversation

zekehuntergreen
Copy link
Contributor

@zekehuntergreen zekehuntergreen commented Oct 3, 2023

Some file types fail to be ingested and ingestion is retried over and over in an infinite loop, inserting new rows into the observability database every few minutes. in #149 we stopped this from happening by moving failed files to a dead-letter bucket.
Any files that have previously been in an infinite loop, and any files that cause this failure mode for other reasons that we haven't thought of in the future should be shown in the observability dashboard.

Before

  • files with more than 100 ingestion events are assumed to be stuck in an infinitely loop and, in order to improve performance, are excluded from the query that selects the data for the dashboard.

After

  • These problematic files are added back into the observability dashboard. In order to avoid slowing down the query, we don't array aggregate the ingestion events' errors (of which there could be thousands).
  • Most of the dropdown in the dashboard for these files is replaced with a message explaining that there's an infinite loop and providing the blob id so that the problem can be investigated
Screenshot 2023-10-04 at 17 49 45

How to test

How can we measure success?

Have we considered potential risks?

Images

Accessibility

@zekehuntergreen zekehuntergreen requested a review from a team as a code owner October 3, 2023 15:55
Copy link
Contributor

@marjisound marjisound left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good 👍

@zekehuntergreen zekehuntergreen merged commit 257da86 into main Oct 9, 2023
1 check passed
@zekehuntergreen zekehuntergreen deleted the show-infinitely-looping-files-in-observability-dashboard branch October 9, 2023 09:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants