Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SEARCH_VIEW] Investigate high memory consumption during requests with many regular expression groups #192

Closed
svlad-90 opened this issue Dec 4, 2023 · 1 comment · Fixed by #201
Assignees
Labels
search_view Issue is related to the [SEARCH_VIEW] domain
Milestone

Comments

@svlad-90
Copy link
Owner

svlad-90 commented Dec 4, 2023

How can one reproduce this issue?

  1. Work with the data set that contains ~ 2M messages ( ~ 512 Mb of data )
  2. Work with the regular expression that makes it possible to create a sequence diagram out of those 2M messages. The "UML" checkbox in the "Search view" should become available for all the found messages, after the search is started.
  3. Start the search
  4. Check the memory consumption. It might increase up to > 10 GB

This is a new issue caused by the creation of sequence diagrams for a large number of messages. The plugin was not used in this way before, so it was expected that something like this might happen.

=> It is not a memory leak, but rather a properly allocated, still huge area of memory, that is used for storing the sequence diagram metadata. At least now we have such an assumption, that should be checked.

Use Valgrind massif to locate the high memory consumption location. Then try to eliminate it.

@svlad-90 svlad-90 added the uml_view Issues related to the uml view label Dec 4, 2023
@svlad-90 svlad-90 added this to the v.1.0.28 milestone Dec 4, 2023
@svlad-90 svlad-90 self-assigned this Dec 4, 2023
@svlad-90 svlad-90 changed the title [uml_view] Investigate high memory consumption of the "uml_view" in case of "too many messages fitting for creation of the sequence diagram" [highlighting] Investigate high memory consumption during requests with many regular expression groups Jun 6, 2024
@svlad-90
Copy link
Owner Author

svlad-90 commented Jun 6, 2024

It was identified that the root cause of this issue is located in the highlighting metadata. Any search query that:

  • finds a lot of messages - millions
  • that contains a lot of captured regular expression groups

will cause a high RAM consumption.

We need to refactor the highlighting functionality to consume less RAM.

@svlad-90 svlad-90 changed the title [highlighting] Investigate high memory consumption during requests with many regular expression groups [HIGHLIGHTING] Investigate high memory consumption during requests with many regular expression groups Jun 6, 2024
@svlad-90 svlad-90 added search_view Issue is related to the [SEARCH_VIEW] domain and removed uml_view Issues related to the uml view labels Jun 6, 2024
@svlad-90 svlad-90 changed the title [HIGHLIGHTING] Investigate high memory consumption during requests with many regular expression groups [SEARCH_VIEW] Investigate high memory consumption during requests with many regular expression groups Jun 6, 2024
@svlad-90 svlad-90 linked a pull request Jun 13, 2024 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
search_view Issue is related to the [SEARCH_VIEW] domain
Projects
Development

Successfully merging a pull request may close this issue.

1 participant