This is the last stable release of FlexFlow before the Unity merge. Unity enables joint optimization of algebraic transformations and parallelization and generally achieves better performance and scalability compared to the original FlexFlow without Unity's optimizations. The Unity merge introduces the following major changes to FlexFlow.
-
With Unity, we now use parallel computation graphs (PCGs) to represent a DNN model. PCG is a unified representation of
distributed DNN training that simultaneously expresses computation, parallelism, and data movement. A detailed description of PCG is available here. -
We add support for Unity's additional forms of parallelism, including reduction parallelism and other operator-specific parallelization strategies.
-
We replace FlexFlow's MCMC search with a three-layer hierarchical search algorithm, which discovers joint optimization of algebraic transformations and parallelization and achieves better performance and scalability compared to FlexFlow's MCMC search.
Starting from this release, Unity's changes will be available in the master branch of the FlexFlow repository.