A Comprehensive Benchmark Suite for Topological Deep Learning

Assess how your model compares against state-of-the-art topological neural networks.

Overview • Get Started • Tutorials • Neural Networks • Liftings • Datasets • References

📌 Overview

TopoBenchmark (TB) is a modular Python library designed to standardize benchmarking and accelerate research in Topological Deep Learning (TDL). In particular, TB allows to train and compare the performances of all sorts of Topological Neural Networks (TNNs) across the different topological domains, where by topological domain we refer to a graph, a simplicial complex, a cellular complex, or a hypergraph. For detailed information, please refer to the TopoBenchmark: A Framework for Benchmarking Topological Deep Learning paper.

The main pipeline trains and evaluates a wide range of state-of-the-art TNNs and Graph Neural Networks (GNNs) (see ⚙️ Neural Networks) on numerous and varied datasets and benchmark tasks (see 📚 Datasets ). Through TopoTune (see 💡 TopoTune), the library provides easy access to training and testing an entire landscape of graph-based TNNs, new or existing, on any topological domain.

Additionally, the library offers the ability to transform, i.e. lift, each dataset from one topological domain to another (see 🚀 Liftings), enabling for the first time an exhaustive inter-domain comparison of TNNs.

🧩 Get Started

Create Environment

If you do not have conda on your machine, please follow their guide to install it.

First, clone the TopoBenchmark repository and set up a conda environment tbx with python 3.11.3.

git clone git@github.com:geometric-intelligence/topobenchmarkx.git
cd TopoBenchmark
conda create -n tbx python=3.11.3

Next, check the CUDA version of your machine:

/usr/local/cuda/bin/nvcc --version

and ensure that it matches the CUDA version specified in the env_setup.sh file (CUDA=cu121 by default). If it does not match, update env_setup.sh accordingly by changing both the CUDA and TORCH environment variables to compatible values as specified on this website.

Next, set up the environment with the following command.

source env_setup.sh

This command installs the TopoBenchmark library and its dependencies.

Run Training Pipeline

Next, train the neural networks by running the following command:

python -m topobenchmarkx 

Thanks to hydra implementation, one can easily override the default experiment configuration through the command line. For instance, the model and dataset can be selected as:

python -m topobenchmarkx model=cell/cwn dataset=graph/MUTAG

Remark: By default, our pipeline identifies the source and destination topological domains, and applies a default lifting between them if required.

The same CLI override mechanism also applies when modifying more finer configurations within a CONFIG GROUP. Please, refer to the official hydradocumentation for further details.

🚲 Experiments Reproducibility

To reproduce Table 1 from the TopoBenchmark: A Framework for Benchmarking Topological Deep Learning paper, please run the following command:

bash scripts/reproduce.sh

Remark: We have additionally provided a public W&B (Weights & Biases) project with logs for the corresponding runs (updated on June 11, 2024).

⚓ Tutorials

Explore our tutorials for further details on how to add new datasets, transforms/liftings, and benchmark tasks.

⚙️ Neural Networks

We list the neural networks trained and evaluated by TopoBenchmark, organized by the topological domain over which they operate: graph, simplicial complex, cellular complex or hypergraph. Many of these neural networks were originally implemented in TopoModelX.

Graphs

Model	Reference
GAT	Graph Attention Networks
GIN	How Powerful are Graph Neural Networks?
GCN	Semi-Supervised Classification with Graph Convolutional Networks

Simplicial complexes

Model	Reference
SAN	Simplicial Attention Neural Networks
SCCN	Efficient Representation Learning for Higher-Order Data with Simplicial Complexes
SCCNN	Convolutional Learning on Simplicial Complexes
SCN	Simplicial Complex Neural Networks

Cellular complexes

Model	Reference
CAN	Cell Attention Network
CCCN	Inspired by A learning algorithm for computational connected cellular network, implementation adapted from Generalized Simplicial Attention Neural Networks
CXN	Cell Complex Neural Networks
CWN	Weisfeiler and Lehman Go Cellular: CW Networks

Hypergraphs

Model	Reference
AllDeepSet	You are AllSet: A Multiset Function Framework for Hypergraph Neural Networks
AllSetTransformer	You are AllSet: A Multiset Function Framework for Hypergraph Neural Networks
EDGNN	Equivariant Hypergraph Diffusion Neural Operators
UniGNN	UniGNN: a Unified Framework for Graph and Hypergraph Neural Networks
UniGNN2	UniGNN: a Unified Framework for Graph and Hypergraph Neural Networks

Combinatorial complexes

Model	Reference
GCCN	Generalized Combinatorial Complex Neural Networks

💡 TopoTune

We include TopoTune, a comprehensive framework for easily defining and training new, general TDL models on any domain using any (graph) neural network ω as a backbone. The pre-print detailing this framework is TopoTune: A Framework for Generalized Combinatorial Complex Neural Networks. In a GCCN (pictured below), the input complex is represented as an ensemble of strictly augmented Hasse graphs, one per neighborhood of the complex. Each of these Hasse graphs is processed by a sub model ω, and the outputs are rank-wise aggregated in between layers.

Defining and training a GCCN

To implement and train a GCCN, run the following command line with the desired choice of dataset, lifting domain (ex: cell, simplicial), PyTorch Geometric backbone model (ex: GCN, GIN, GAT, GraphSAGE) and parameters (ex. model.backbone.GNN.num_layers=2), neighborhood structure (routes), and other hyperparameters.

python -m topobenchmarkx \
    dataset=graph/PROTEINS \
    dataset.split_params.data_seed=1 \
    model=cell/topotune\
    model.tune_gnn=GCN \
    model.backbone.GNN.num_layers=2 \
    model.backbone.routes=\[\[\[0,0\],up_laplacian\],\[\[2,1\],boundary\]\] \
    model.backbone.layers=4 \
    model.feature_encoder.out_channels=32 \
    model.feature_encoder.proj_dropout=0.3 \
    model.readout.readout_name=PropagateSignalDown \
    logger.wandb.project=TopoTune_cell \
    trainer.max_epochs=1000 \
    callbacks.early_stopping.patience=50 \

To use a single augmented Hasse graph expansion, use model={domain}/topotune_onehasse instead of model={domain}/topotune.

To specify a set of neighborhoods (routes) on the complex, use a list of neighborhoods each specified as \[\[{source_rank}, {destination_rank}\], {neighborhood}\]. Currently, the following options for {neighborhood} are supported:

• up_laplacian, from rank $r$ to $r$
• down_laplacian, from rank $r$ to $r$
• boundary, from rank $r$ to $r-1$
• coboundary, from rank $r$ to $r+1$
• adjacency, from rank $r$ to $r$ (stand-in for up_adjacency, as down_adjacency not yet supported in TopoBenchmark)

Using backbone models from any package

By default, backbone models are imported from torch_geometric.nn.models. To import and specify a backbone model from any other package, such as torch.nn.Transformer or dgl.nn.GATConv, it is sufficient to 1) make sure the package is installed and 2) specify in the command line:

model.tune_gnn = {backbone_model}
model.backbone.GNN._target_={package}.{backbone_model}

Reproducing experiments

We provide scripts to reproduce experiments on a broad class of GCCNs in scripts/topotune and reproduce iterations of existing neural networks in scripts/topotune/existing_models, as previously reported in the TopoTune paper.

We invite users interested in running extensive sweeps on new GCCNs to replicate the --multirun flag in the scripts. This is a shortcut for running every possible combination of the specified parameters in a single command.

🚀 Liftings

We list the liftings used in TopoBenchmark to transform datasets. Here, a lifting refers to a function that transforms a dataset defined on a topological domain (e.g., on a graph) into the same dataset but supported on a different topological domain (e.g., on a simplicial complex).

Topology Liftings

Graph2Simplicial

Name	Description	Reference
CliqueLifting	The algorithm finds the cliques in the graph and creates simplices. Given a clique the first simplex added is the one containing all the nodes of the clique, then the simplices composed of all the possible combinations with one node missing, then two nodes missing, and so on, until all the possible pairs are added. Then the method moves to the next clique.	Simplicial Complexes
KHopLifting	For each node in the graph, take the set of its neighbors, up to k distance, and the node itself. These sets are then treated as simplices. The dimension of each simplex depends on the degree of the nodes. For example, a node with d neighbors forms a d-simplex.	Neighborhood Complexes

Graph2Cell

Name	Description	Reference
CellCycleLifting	To lift a graph to a cell complex (CC) we proceed as follows. First, we identify a finite set of cycles (closed loops) within the graph. Second, each identified cycle in the graph is associated to a 2-cell, such that the boundary of the 2-cell is the cycle. The nodes and edges of the cell complex are inherited from the graph.	Appendix B

Graph2Hypergraph

Name	Description	Reference
KHopLifting	For each node in the graph, the algorithm finds the set of nodes that are at most k connections away from the initial node. This set is then used to create an hyperedge. The process is repeated for all nodes in the graph.	Section 3.4
KNearestNeighborsLifting	For each node in the graph, the method finds the k nearest nodes by using the Euclidean distance between the vectors of features. The set of k nodes found is considered as an hyperedge. The proces is repeated for all nodes in the graph.	Section 3.1

Feature Liftings

Name	Description	Supported Domains
ProjectionSum	Projects r-cell features of a graph to r+1-cell structures utilizing incidence matrices (B_{r}).	Simplicial, Cell
ConcatenationLifting	Concatenate r-cell features to obtain r+1-cell features.	Simplicial

Data Transformations

Transform	Description	Reference
Message Passing Homophily	Higher-order homophily measure for hypergraphs	Source
Group Homophily	Higher-order homophily measure for hypergraphs that considers groups of predefined sizes	Source

📚 Datasets

Dataset	Task	Description	Reference
Cora	Classification	Cocitation dataset.	Source
Citeseer	Classification	Cocitation dataset.	Source
Pubmed	Classification	Cocitation dataset.	Source
MUTAG	Classification	Graph-level classification.	Source
PROTEINS	Classification	Graph-level classification.	Source
NCI1	Classification	Graph-level classification.	Source
NCI109	Classification	Graph-level classification.	Source
IMDB-BIN	Classification	Graph-level classification.	Source
IMDB-MUL	Classification	Graph-level classification.	Source
REDDIT	Classification	Graph-level classification.	Source
Amazon	Classification	Heterophilic dataset.	Source
Minesweeper	Classification	Heterophilic dataset.	Source
Empire	Classification	Heterophilic dataset.	Source
Tolokers	Classification	Heterophilic dataset.	Source
US-county-demos	Regression	In turn each node attribute is used as the target label.	Source
ZINC	Regression	Graph-level regression.	Source

🛠️ Development

To join the development of TopoBenchmark, you should install the library in dev mode.

For this, you can create an environment using either conda or docker. Both options are detailed below.

🐍 Using Conda Environment

Follow the steps in 🧩 Get Started.

🐳 Using Docker

For ease of use, TopoBenchmark employs Docker. To set it up on your system you can follow their guide. once installed, please follow the next steps:

First, clone the repository and navigate to the correct folder.

git clone git@github.com:geometric-intelligence/topobenchmarkx.git
cd TopoBenchmark

Then, build the Docker image.

docker build -t topobenchmarkx:new .

Depending if you want to use GPUs or not, these are the commands to run the Docker image and mount the current directory.

With GPUs

docker run -it -d --gpus all --volume $(pwd):/TopoBenchmark topobenchmarkx:new

With CPU

docker run -it -d --volume $(pwd):/TopoBenchmark topobenchmarkx:new

Happy development!

🔍 References

To learn more about TopoBenchmark, we invite you to read the paper:

@article{telyatnikov2024topobenchmark,
      title={TopoBenchmark: A Framework for Benchmarking Topological Deep Learning}, 
      author={Lev Telyatnikov and Guillermo Bernardez and Marco Montagna and Pavlo Vasylenko and Ghada Zamzmi and Mustafa Hajij and Michael T Schaub and Nina Miolane and Simone Scardapane and Theodore Papamarkou},
      year={2024},
      eprint={2406.06642},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2406.06642}, 
}

If you find TopoBenchmark useful, we would appreciate if you cite us!

🐭 Additional Details

Hierarchy of configuration files

├── configs                   <- Hydra configs
│   ├── callbacks                <- Callbacks configs
│   ├── dataset                  <- Dataset configs
│   │   ├── graph                    <- Graph dataset configs
│   │   ├── hypergraph               <- Hypergraph dataset configs
│   │   └── simplicial               <- Simplicial dataset configs
│   ├── debug                    <- Debugging configs
│   ├── evaluator                <- Evaluator configs
│   ├── experiment               <- Experiment configs
│   ├── extras                   <- Extra utilities configs
│   ├── hparams_search           <- Hyperparameter search configs
│   ├── hydra                    <- Hydra configs
│   ├── local                    <- Local configs
│   ├── logger                   <- Logger configs
│   ├── loss                     <- Loss function configs
│   ├── model                    <- Model configs
│   │   ├── cell                     <- Cell model configs
│   │   ├── graph                    <- Graph model configs
│   │   ├── hypergraph               <- Hypergraph model configs
│   │   └── simplicial               <- Simplicial model configs
│   ├── optimizer                <- Optimizer configs
│   ├── paths                    <- Project paths configs
│   ├── scheduler                <- Scheduler configs
│   ├── trainer                  <- Trainer configs
│   ├── transforms               <- Data transformation configs
│   │   ├── data_manipulations       <- Data manipulation transforms
│   │   ├── dataset_defaults         <- Default dataset transforms
│   │   ├── feature_liftings         <- Feature lifting transforms
│   │   └── liftings                 <- Lifting transforms
│   │       ├── graph2cell               <- Graph to cell lifting transforms
│   │       ├── graph2hypergraph         <- Graph to hypergraph lifting transforms
│   │       ├── graph2simplicial         <- Graph to simplicial lifting transforms
│   │       ├── graph2cell_default.yaml  <- Default graph to cell lifting config
│   │       ├── graph2hypergraph_default.yaml <- Default graph to hypergraph lifting config
│   │       ├── graph2simplicial_default.yaml <- Default graph to simplicial lifting config
│   │       ├── no_lifting.yaml           <- No lifting config
│   │       ├── custom_example.yaml       <- Custom example transform config
│   │       └── no_transform.yaml         <- No transform config
│   ├── wandb_sweep              <- Weights & Biases sweep configs
│   │
│   ├── __init__.py              <- Init file for configs module
│   └── run.yaml               <- Main config for training

More information regarding Topological Deep Learning

Topological Graph Signal Compression

Architectures of Topological Deep Learning: A Survey on Topological Neural Networks

TopoX: a suite of Python packages for machine learning on topological domains