ddl-benchmarks: Benchmarks for Distributed Deep Learning.

Introduction

This repository contains a set of benchmarking scripts for evaluating the training performance of popular distributed deep learning methods in our paper, which mainly focuses on system-level optimization algorithms of synchronized stochastic gradient descent with data parallelism. Currently, it covers:

system architectures

• Parameter server with BytePS.
• All-to-all with Horovod.

optimization algorithms

• Wait-free backpropagation (WFBP), which is also known as the technique of pipelining the backward computations with gradient communications and it is a default feature in current deep learning frameworks.
• Tensor fusion, which has been integraded in Horovod with a hand-craft threshold to determine when to fuse tensors, but it is possible to dynamically determine to fuse tensors in MG-WFBP.
• Tensor partition and priority schedule, which are proposed in ByteScheduler.
• Gradient compression with quantization (i.e., signSGD) and sparsification (i.e., TopK-SGD). These methods are included in the code, but they are excluded from our paper as the paper focuses on the system-level optimization methods.

deep neural networks

• Convolutional neural networks (CNNs) on a fake ImageNet data set (i.e., randomly generate the input image of 224*224*3)
• Transformers: BERT-Base and BERT-Large pretraining models.

Installation

Prerequisites

• Python 3.6+
• CUDA-10.+
• NCCL-2.4.+
• PyTorch-1.4.+
• OpenMPI-4.0.+
• Horovod-0.19.+
• BytePS-0.2.+
• ByteScheduler
• bit2byte: Optional if not run signSGD.

Get the code

$git clone https://github.com/HKBU-HPML/ddl-benchmarks.git
$cd ddl-benchmarks 
$pip install -r requirements.txt

Configure the cluster settings

Before running the scripts, please carefully configure the configuration files in the directory of configs.

• configs/cluster*: configure the host files for MPI
• configs/envs.conf: configure the cluster enviroments.

Create a log folder, e.g.,

$mkdir -p logs/pcie

Run benchmarks

• The batch mode

$python benchmarks.py

• The individual mode, e.g.,

$cd horovod
$dnn=resnet50 bs=64 nworkers=64 ./horovod_mpi_cj.sh

Paper

If you are using this repository for your paper, please cite our work

@article{shi2020ddlsurvey,
    author = {Shi, Shaohuai and Tang, Zhenheng and Chu, Xiaowen and Liu, Chengjian and Wang, Wei and Li, Bo},
    title = {Communication-Efficient Distributed Deep Learning: Survey, Evaluation, and Challenges},
    journal = {arXiv},
    url = {\url{https://arxiv.org/pdf/2005.13247.pdf}},
    year = {2020}
}