Benchmarking of search engines with the ground truth of spatial proteomics datasets

[JuliaS92]

Title

Benchmarking of search engines with the ground truth of spatial proteomics datasets

Abstract

Benchmarks of search engines are crucial for selecting optimal tools in computational proteomics. Current benchmarks typically assess depth, coefficient of variation, and accuracy in mixed species experiments. While these mixed species benchmarks represent significant progress in software evaluation, they address only one of many use cases in proteomics and diverge from more common single-species experiments. Establishing a testable ground truth in real-life datasets remains challenging. However, spatial proteomics and SEC-MS experiments offer an inherent biochemical ground truth, as members of bona fide protein complexes exhibit near identical profiles. We propose leveraging this principle for software benchmarking, building upon previous work with dynamic organellar maps. By combining established measures with a carefully selected set of reference datasets, we aim to develop a comprehensive Proteobench module. This will provide an additional software benchmark that specifically addresses single-species performance and usability in profiling experiments, thereby enhancing the evaluation of proteomics software tools.

Project Plan

The project comprises three major tasks:

1. Adapting the exsting code from DOM-ABC (github, hosted GUI, paper) that provides the processing workflow and benchmarking measures into a Proteobench module. The code is python based, so no translations are necessary.
2. Selection of reference datasets for different experimental formats and acquisition schemes. This could initially be the datasets from the paper, but can be expanded for example to SEC-MS datasets and TMT based hyperLOPIT maps.
3. Running the selected datasets through search engines and loading the results into proteobench to have a starting point for continuous software benchmarking.

If this gets selected the DOM-ABC github repository will be optimized for a pypi submission with few dependencies prior to the hackathon.

During the hackathon we will therefore:

1. Set up a new Proteobench fork for this project and create a new module interface to collect parameters.
2. Integrate the DOM-ABC workflow into the module.
3. Decide which protein complexes to use in the benchmarking and which measures to store in the database.
4. Select at least one, ideally several datasets for benchmarking to cover a range of acquisition modes and experimental complexities.
5. Run the datasets through different search engines to create initial results files.
6. Test the module using the generated results.

After the hackaton we will ideally have generated a fully functional Proteobench module, that can be pulled by the core team. Otherwise the project will be finished by anybody interested after the hackathon. If the project is successful and yields an interesting perspective on search engine benchmarking, a short paper could be published with all participants of the hackathon.

Technical Details

Programming language: python
Will build on existing software: Proteobench, DOM-ABC
These datasets will be used:
DDA DOMS: https://www.ebi.ac.uk/pride/archive/projects/PXD034962
DIA DOMS: https://www.ebi.ac.uk/pride/archive/projects/PXD034971

Contact information

Julia Schessner
Max Planck Institute for Biochemistry
Department for Proteomics and Signal Transduction
schessner@biochem.mpg.de