Output benchmarks in machine-readable format

[scotttrinh]

I use the @arktest/attest package extensively for iterating on expensive types in https://github.com/edgedb/edgedb-js, and I always find myself manually parsing the output to generate visualizations and to update the benchmarks themselves. I was thinking about building some automation around these things, but the current output is not easy to parse.

I believe the most flexible output format would be some structured JSON, so I'm opening up this discussion to gather some feedback about some possible schemas for this output.

[scotttrinh]

Here's one possible output that is roughly based on the existing text for type instantiation benchmarks:

interface BenchmarkTypeInstantiationResult {
  name: string;
  baseline: number;
  result: number;
}

It might be useful to provide some computed data in the result as well, but maybe minimal is a good place to start.

[ssalbdivad]

There is no historical format anymore, it was something we experimented with for a while where whenever a benchmark was updated, the name, data and results would be added to a JSON file so changes could be tracked over time.

I do think that would be a nice feature but I think just starting with outputting JSON results for a single run would make more sense.

[scotttrinh]

here is no historical format anymore, it was something we experimented with for a while where whenever a benchmark was updated, the name, data and results would be added to a JSON file so changes could be tracked over time.

Ahh, yes, this is exactly the use case I want to build out myself for doing visualizations! I think that idea can be decoupled from this effort and probably from this project altogether. I think it's worth thinking about the output through that lens though: whatever we output here is something that will feed downstream tooling. For my own visualizations, I've really just used the { name, result } pair and plotted the value of result over different changes, grouped by name.

starting with outputting JSON results for a single run would make more sense.

👍

[scotttrinh]

Maybe we need to capture the "what" of the measurement here as well rather than having each measurement be a separate interface?

So something like:

interface BaseBenchResult {
  name: string;
  baseline: number;
  result: number;
}

interface TypeBenchResult extends BaseBenchResult {
  _tag: "TypeBenchResult";
  unit: TypeUnit;
}

interface TimeBenchResult extends BaseBenchResult {
  _tag: "TimeBenchResult";
  unit: TimeUnit;
}

—?

[ssalbdivad]

This looks good to me. The only change I'd make is renaming _tag to something like kind and having the values be "static" and "runtime" so it's a bit more friendly to use as an external property.

[scotttrinh]

In its somewhat final form based on suggestions:

interface BaseBenchResult {
  name: string;
  baseline: number;
  result: number;
}

interface StaticBenchResult extends BaseBenchResult {
  kind: "static";
  unit: TypeUnit;
}

interface RuntimeBenchResult extends BaseBenchResult {
  kind: "runtime";
  unit: TimeUnit;
}