Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add interactive website via Github Pages #46

Merged
merged 29 commits into from
Jul 14, 2023
Merged

Conversation

caibear
Copy link
Contributor

@caibear caibear commented Jul 6, 2023

This pull request adds an interactive website. Given bandwidth and cpu limits, it calculates how many messages per second could be sent/received for different combinations of serialization crates and compression libraries.

See https://caibear.github.io/rust_serialization_benchmark/

For example, this is useful for calculating how many average concurrent players an mk48.io server can handle. Given inputs 1 TB/Mo and 0.01 cores, it returns 437 updates/s for bitcode. SInce mk48.io sends 10 updates/s per player, a server can handle 43.7 players. The second best is serde_bare + zstd which returns 387 updates/s aka 38.7 players.

The data is taken from a copy of the README.md embedded in the binary. Compression speeds are currently based on constants, ideally they would be measured during the benchmarks.

TODO

  • Add link in README.md
  • Add Cargo.lock?

@djkoloski
Copy link
Owner

Thanks for the PR, I think this is very exciting for the benchmarks. I'll break down my thought process a bit:

Direction

  • Hosting a web page with all of the relevant information in one place is the right idea.
  • The benchmarks would benefit greatly from visualization, especially interactive visualization.
  • As with all additions to the benchmark, we must strive to ensure that the estimation and presentation are as accurate, precise, and neutral as possible. When we can't achieve those goals, we should present all options and be maximally descriptive.

Technical

  • The benchmark parsing code is old, we should just replace it with some Rust.
  • Storing the raw benchmark logs is nice, but we should also process them into JSON or another portable format that we can easily work with.
  • We should run the full benchmark suite multiple times to get data for a range of input sizes. The benchmarks currently prefer XL data sets because it more accurately measures the performance relative to input size. For smaller input sizes, one-time overhead may dominate and that should be quantified and measured.

With that in mind, here's what I propose:

  1. Convert the benchmark parsing code to Rust. It should parse a raw log and output a JSON file.
  2. Switch the README formatter to consume the JSON file. This can also be converted to Rust.
  3. Switch this interactive site to consume that JSON file.
  4. Document the calculations used to make the projections in this visualizer.

With future work to add more input sizes to the benchmark data sets.

@caibear I would appreciate your feedback and thoughts. I understand this is probably a significant expansion of the intended scope, so I would of course help get this work done.

.github/workflows/bench.yml Outdated Show resolved Hide resolved
pages/Cargo.toml Outdated Show resolved Hide resolved
pages/src/calc.rs Outdated Show resolved Hide resolved

pub fn serialize_seconds(self, bytes: u64) -> f32 {
// TODO real benchmarks (since speed is different on different data and cpus).
const SCALE: f32 = 0.5;
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is this constant for?

@djkoloski djkoloski merged commit b0ae613 into djkoloski:master Jul 14, 2023
@caibear caibear deleted the calc branch March 24, 2024 06:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants