A module to plot ITK performance benchmarking data.
Three dataset are provided for the purpose of demonstrating the visualization capabilities of the module:
- simplified_data.tar.gz: a reduced set of files, primarily intended as a restricted benchmarking dataset.
- scatter_os_data.tar.gz: a dataset that is intended to demonstrate the capabilities of the scatter plot visualization method, and provide a means to compare the performance of a module across different platforms and ITK versions.
- clay_data.tar.gz: original
claybenchmarking files hosted at data.kitware.com, a dataset intended to fully exercise the module.
Note that the JSON files in the
simplified_data.tar.gz and the
scatter_os_data.tar.gz were
manually edited from the original clay
benchmarking files
hosted in the data.kitware.com to simulate
benchmarks for ITK versions that were not present in the original clay
dataset, as well as to give rise to some situations (e.g. modules not present in
some ITK versions, different number of time probes across modules or versions,
etc.) that could potentially arise when benchmarking ITK modules across time.
Thus, they may not reflect actual ITK module performances.
The visualization scripts were tested on all three datasets using Python 3.6.5.
Please follow the steps described:
Clone the repository.
Install Python.
Install the requirements using the requirements.txt file provided.
Run the example visualization program on the input dataset by providing the folder where the (uncompressed) benchmarking
JSONfiles are located, and provide a filename for theJSONsummary, e.g.:python ./examples/itk_performance_benchmarking.py ./example_data/clay ./output/summary.json
These example visualizations were created using the JSON files contained in
the scatter_os_data.tar.gz example
dataset.
Module error bar plot:
Module scatter plot:
Historical heatmap:
This software is distributed under the Apache 2.0 license. Please see the LICENSE file for details.