dcp

distributed copy

mpiFileUtils ships with 2 versions of this tool: dcp and dcp1.

dcp1 is based off of the original dcp implementation. It uses libcircle to simultaneously walk and copy data. This algorithm works very well for large directory trees with lots of small files. However, we found this approach could lead to high write lock contention when copying large files on Lustre, as the data blocks within a file can be randomly distributed to different processes.

To reduce the write lock contention, we created the new dcp. This breaks the walk and copy operations into two distinct phases. It uses libcircle to first walk everything and builds up a list of files to be copied. After the walk completes, it then statically assigns data blocks to files to and starts to copy. Although the work assignment is static, we know the full set of bytes to be copied, so we can better assign work to processes to avoid excessive Lustre write lock contention.

Since both algorithms are useful at different times, our plan is to merge both algorithms into one tool (dcp). Ideally, the tool would select the algorithm to be used dynamically, or if that’s too hard, we could have an option that lets the user declare which algorithm to use. Since we’ve not yet done this merge, we have kept the original pure libcircle-based algorithm handy in its own tool, dcp1.

Depending on your workload, you may find either dcp or dcp1 to be the faster option. We generally recommend using dcp.