Increased memory consumption with VegaFusion

[ericmassip]

Our Django app uses Altair to generate graphs and display them on the browser. A couple months ago, one of our graphs exceeded the 5000 rows limit and we got the famous MaxRowsErrors. As suggested by the Altair docs, we installed VegaFusion (using version 1.6.1).

The migration was very smooth, the graphs are generated and displayed correctly. The problem is that we've been experiencing an increase in the memory consumption of our Django processes of ~100/150MBs. It doesn't go further than that, but we're wondering if that increase is expected or we should do something differently. Thx!

[jonmmease]

Hi @ericmassip, glad to hear the migration to VegaFusion went smoothly! As long as the memory usage doesn't continue increasing, this is expected.

Is this causing issues in your application?

[ericmassip]

Thanks @jonmmease for your quick response! Glad to hear that! :D

It's not causing issues but our production server is a bit short on memory right now, that's why we can "feel" any memory spike that occurs. We're planning to upgrade it but we wanted to make sure that we were not underestimating anything. Basically, we wouldn't want to upgrade our servers and accept an increase in memory consumption without being sure.

[intiocean]

Thanks @jonmmease - good to know that ~150MB is the expected memory usage of using VegaFusion.

As @ericmassip mentioned. We were surprised, as we have a number of workers serving our web content (in standard Django style) and their memory footprint very quickly doubled (as they have a normal footprint of ~150MB)!

We'll have to think about if it's worth us running the VegaFusion and increasing our memory requirement for each web worker. I think ours is probably a use case that can get away without needing it. I can see this is much less of an issue if you are only running one process and so only have the memory bump up once.

[jonmmease]

Out of curiosity, how many web workers do you run per server?

One future possibility, that isn't fully fleshed out yet, is to run a single instance of VegaFusion server (which is a standalone executable that serves VegaFusion's features over gRPC) and have all of the Python processes connect to that. If this is something that would be of interest I can look into it more.

[intiocean]

We have a small number of workers: 3 per application and we have 2 applications running off the same codebase so 6 workers at present. NB: This is a very low traffic subscribers only access site.

Thank you for sharing your thoughts about running a single standalone instance.
For now, we have a very simple setup (with few external services) so a VegaFusion standalone server wouldn't appeal to us at the moment either. If we had a greater use of charts where it was really helping us then it would be more likely to be worth the extra complexity. It sounds like a sensible way forward in those cases but that isn't us at the moment.

[jonmmease]

Migrated to a discussion as it's not directly actionable at the moment. Let me know if you get to the point where the separate server architecture would be appealing.