Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make Cray XC builds work out of the box #268

Open
wants to merge 11 commits into
base: master
Choose a base branch
from
59 changes: 47 additions & 12 deletions BUILD.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,8 @@ This will build the Grappa static library (in `build/Make+Release/system/libGrap
# in `build/Make+Release`
make graph_new.exe
# generates <project_root>/build/Make+Release/applications/graph500/grappa/graph_new.exe
# to run, use the 'srun' script which has been copied to build/Make+Release/bin:
bin/grappa_srun --nnode=4 --ppn=4 -- applications/graph500/grappa/graph_new.exe --scale=20 --bench=bfs
# now run as you would any other MPI job. If your cluster uses Slurm, do:
srun --nodes=4 --ntasks-per-node=4 -- applications/graph500/grappa/graph_new.exe --scale=20 --bench=bfs

This is the simplest build configuration. You are likely to want to specify more things, such as a specific compiler, a directory for already-installed dependencies so you don't have to rebuild them for each new configuration, and more. So read on.

Expand All @@ -40,6 +40,7 @@ You must have a Linux system with the following installed to be able to build Gr
* MVAPICH2 >= 1.9
* MPICH >= 3.1
* Intel MPI >= 5.0.2.044
* Cray MPT >= 6.0, probably (tested with 7.1.3)

The following dependencies are dealt with automatically. You may want to override the default behavior for your specific system as described in the next section, especially in the case of Boost (if you already have a copy in a non-standard place).

Expand Down Expand Up @@ -69,23 +70,33 @@ The `configure` script creates a new "build/*" subdirectory and runs CMake to ge
Default: Make.
Can specify multiple with commas.
--mode=[Release]|Debug[,*] Build mode. Default: Release (with debug symbols)
--cc=path/to/c/compiler Use alternative C compiler.
--cc=path/to/c/compiler Use alternative C compiler (infer C++ compiler from this path).
--cxx=path/to/c++/compiler Use alternative C++ compiler (infer C compiler from this path).
--boost=path/to/boost/root Specify location of compatible boost (>= 1.53)
(otherwise, cmake will download and build it)
--name=NAME Add an additional name to this configuration to distinguish it
(i.e. compiler version)
--tracing Enable VampirTrace/gperftools-based sampled tracing. Looks for
VampirTrace build in 'third-party' dir.
--name=NAME Add an additional name to this configuration to
distinguish it (i.e. compiler version)
--tracing Enable VampirTrace/gperftools-based sampled tracing.
Looks for VampirTrace build in 'third-party' dir.
--profiling Enable gperftools-based sampled profiling.
--vampir=path/to/vampirtrace/root
Specify path to VampirTrace build (enables tracing).
--third-party=path/to/built/deps/root
Can optionally pre-build third-party dependencies instead of
re-building for each configuration.
Can optionally pre-build third-party dependencies
instead of re-building for each configuration.
--third-party-cc=path/to/c/compiler
Use alternative C compiler just for third-party builds.
--third-party-cxx=path/to/c++/compiler
Use alternative C++ compiler just for third-party builds.
--build-here Just configure into current directory rather than creating new build directories.
--third-party-tarfile=/path/to/file.tar
Instead of downloading third-party dependences from the web, extract them from the specified tar file (available from Grappa website).
--prefix=path/to/grappa/installation
--enable_rpath Set RPATH on binaries to help them find necessary libraries.
--verbose Output verbose configure information.
--prefix=path/to/installed/grappa
Specify destination for Grappa installation (defaults to inside build directory).


To build, after calling `configure`, cd into the generated directory, and use the build tool selected (e.g. `make` or `ninja`), specifying the desired target (e.g. `graph_new.exe` to build the new Graph500 implementation, or `check-New_delegate_tests` to run the delegate tests, or `demo-gups.exe` to build the GUPS demo).

### Generators
Expand All @@ -107,14 +118,30 @@ CMake will download and build `gflags`, `boost`, and `gperfools`. It will build

The external dependencies can be shared between Grappa configurations. If you specify a directory to `--third-party`, CMake will build and install the dependencies there, and then any other configurations will reuse them. Sometimes this won't work; for instance, if using two different compilers, you may have difficulty sharing a third-party directory. If this happens, just make a new third-party directory and rebuild them using the new configuration, or don't specify it and have this configuration build them just for itself.

Because Boost takes the longest to compile and is often included in systems, Boost can be specified separately from the other third-party installs. Existing system installs of the other dependencies should typically *not* be relied on.
Because Boost takes the longest to compile and is often included in systems, Boost can be specified separately from the other third-party installs. You can use the ```--boost``` argument to ```configure``` to point at an existing Boost installation.

Existing system installs of the other dependencies should typically *not* be relied on.

### No web access for third-party dependencies

If you want to build Grappa on a machine without access to the web, and that machine doesn't already have all the third-party libraries installed that Grappa needs, you'll have to provide the source archives for those dependences yourself.

To do so, download this file: [http://grappa.cs.washington.edu/files/grappa-third-party-downloads.tar](http://grappa.cs.washington.edu/files/grappa-third-party-downloads.tar). Then run ```configure``` with the option ```--third-party-tarfile=</path/to/file.tar>``` pointing at the tarfile.

### Building on a Cray XC

Cray provides helpful compiler wrapper scripts (```cc``` and ```CC```) that cause problems with some of the third-party dependences, so the ```configure``` script allows you to specify separate compilers for the dependences with the ```--third-party-cc``` and ```-third-party-cxx``` flags. Since the Cray C++ compiler is not supported, you'll need to use GCC or another supported compiler instead. Here's an example of how to build on a Cray XC:

```
module load cmake ruby
module swap PrgEnv-cray PrgEnv-gnu
./configure --cc=`which cc` --cxx=`which CC` --third-party-cc=`which gcc` --third-party-cxx=`which g++`
cd build/Make+Release
make -j16 && make install
```

Note that due to problems with dynamic linking, Grappa's unit tests don't build properly on a Cray XC yet.

## Installing Grappa

Grappa supports a ```install``` target which will install compiled libraries and header files to a directory on your system. The default install path is an ```install``` directory under the build directory, like ```build/Make+Release/install```. If you'd prefer it to go somewhere else, specify the ```--prefix=<path>``` option when you run ```configure```.
Expand Down Expand Up @@ -169,6 +196,14 @@ We've made it easier to use Grappa as a library by providing a GNU Make include

An example of the first usage is included in the directory ```applications/demos/standalone```.

## Finding libraries; RPATH; RUNPATH

A standard problem when running MPI jobs is making sure the binary is able to find all the shared libraries it needs. Some systems are good about propagating the user's environment to all the job's processes (we've had good experiences with Slurm); but some require setting environment variables like ```LD_LIBRARY_PATH``` manually, which quickly gets complicated. Ideally we'd just build all binaries as static, but some MPI distributions make this difficult.

To make this easier, you can pass the argument ```--enable_rpath``` to the configure script, and Grappa will be configured to set the ```RPATH``` search path in binaries it builds. This is not a foolproof solution. In particular, it overrides any ```LD_LIBRARY_PATH``` variable set in the environment. For this reason, this feature is disabled by default.

If your binaries have trouble finding libraries when you run them, try reconfiguring/rebuilding with ```--enable_rpath```.

## CMake Notes
A couple notes about adding new targets for CMake to build. First: each directory where something is built should typically have a `CMakeLists.txt` file. Somewhere up the directory hierarchy, this directory must be 'added'. For instance, applications directories are added from `applications/CMakeLists.txt`:

Expand Down Expand Up @@ -212,7 +247,7 @@ To make it easy to prototype ideas, there's a directory in root: `scratch`. Any
make rebuild_cache
# then...
make scratch-test.exe
bin/grappa_srun --nnode=2 --ppn=1 -- scratch/scratch-test.exe
srun --nodes=2 --ntasks-per-node=1 -- scratch/scratch-test.exe

### Demos
Similar to the scratch directory, all sub-directories in the `applications/demos` directory will be searched for `.cpp` files, and added as targets (`demo-#{base_name}.exe`). (note: search is not recursive, just one level of subdirectory).
Expand Down
161 changes: 96 additions & 65 deletions CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@ include(util/grappa.cmake)
option(TRACING "Sample statistics with VTrace at regular intervals." OFF)
option(PROFILING "Sample profile with GPerftools at regular intervals." OFF)

option(ENABLE_RPATH "Set RPATH when building binaries." OFF)

SET(GRAPPA_INSTALL_PREFIX "${CMAKE_BINARY_DIR}/install" CACHE PATH "Prefix prepended to install directories")
SET(CMAKE_INSTALL_PREFIX "${GRAPPA_INSTALL_PREFIX}" CACHE INTERNAL "Prefix prepended to install directories" FORCE)

Expand Down Expand Up @@ -44,32 +46,6 @@ else()
message(WARNING "You are using an unsupported compiler! Compilation has only been tested with Clang and GCC.")
endif()

###########################
# Use RUNPATH if available
# use, i.e. don't skip the full RPATH for the build tree
set(CMAKE_SKIP_BUILD_RPATH FALSE)
# use final INSTALL_RPATH even in build tree (this lets us manually add things to CMAKE_INSTALL_RPATH)
set(CMAKE_BUILD_WITH_INSTALL_RPATH TRUE)
set(CMAKE_INSTALL_RPATH "${CMAKE_INSTALL_PREFIX}/lib:${CMAKE_INSTALL_RPATH}")
# add the automatically determined parts of the RPATH
# which point to directories outside the build tree to the install RPATH
# set(CMAKE_INSTALL_RPATH_USE_LINK_PATH TRUE)
set(CMAKE_INSTALL_RPATH_USE_LINK_PATH FALSE)

# the RPATH to be used when installing, but only if it's not a system directory
list(FIND CMAKE_PLATFORM_IMPLICIT_LINK_DIRECTORIES "${CMAKE_INSTALL_PREFIX}/lib" isSystemDir)
if("${isSystemDir}" STREQUAL "-1") # not a system directory
set(CMAKE_INSTALL_RPATH "${CMAKE_INSTALL_RPATH}:${CMAKE_INSTALL_PREFIX}/lib")
endif("${isSystemDir}" STREQUAL "-1")

## Not setting runpath because having MPI libs in LD_LIBRARY_PATH was messing up VampirTrace's ability to find its own libs. Maybe there's another way to fix this, but it just seems more robust (for now) to not allow LD_LIBRARY_PATH to affect our libs (after configure uses it to find them).
# set runpath, too
# if(NOT APPLE)
# set(CMAKE_EXE_LINKER_FLAGS "-Wl,--enable-new-dtags")
# endif()

##### </RUNPATH> ########


#############################################################################
# configure profiling and tracing
Expand Down Expand Up @@ -100,7 +76,9 @@ if(TRACING)
# set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} --as-needed --whole-archive -lvt --no-whole-archive --no-as-needed -lopen-trace-format -lz -lpapi -ldl -lm")

if(NOT THIRD_PARTY_ROOT STREQUAL ${VAMPIR_ROOT})
set(CMAKE_INSTALL_RPATH "${VAMPIR_ROOT}/lib:${CMAKE_INSTALL_RPATH}")
if(ENABLE_RPATH)
set(CMAKE_INSTALL_RPATH "${VAMPIR_ROOT}/lib:${CMAKE_INSTALL_RPATH}")
endif()
endif()
endif()

Expand Down Expand Up @@ -132,7 +110,10 @@ else()
endif()
include_directories("${THIRD_PARTY_ROOT}/include")
link_directories("${THIRD_PARTY_ROOT}/lib")
set(CMAKE_INSTALL_RPATH "${THIRD_PARTY_ROOT}/lib:${CMAKE_INSTALL_RPATH}")

if(ENABLE_RPATH)
set(CMAKE_INSTALL_RPATH "${THIRD_PARTY_ROOT}/lib:${CMAKE_INSTALL_RPATH}")
endif()


#
Expand Down Expand Up @@ -356,12 +337,44 @@ endif()
add_subdirectory(third-party)


# RPATH causes problems on some systems, but is helpful on others. Make it an option
if(ENABLE_RPATH)
###########################
# Use RUNPATH if available
# use, i.e. don't skip the full RPATH for the build tree
set(CMAKE_SKIP_BUILD_RPATH FALSE)
# use final INSTALL_RPATH even in build tree (this lets us manually add things to CMAKE_INSTALL_RPATH)
set(CMAKE_BUILD_WITH_INSTALL_RPATH TRUE)
set(CMAKE_INSTALL_RPATH "${CMAKE_INSTALL_RPATH}:${CMAKE_INSTALL_PREFIX}/lib")
# add the automatically determined parts of the RPATH
# which point to directories outside the build tree to the install RPATH
# set(CMAKE_INSTALL_RPATH_USE_LINK_PATH TRUE)
set(CMAKE_INSTALL_RPATH_USE_LINK_PATH FALSE)

# the RPATH to be used when installing, but only if it's not a system directory
list(FIND CMAKE_PLATFORM_IMPLICIT_LINK_DIRECTORIES "${CMAKE_INSTALL_PREFIX}/lib" isSystemDir)
if("${isSystemDir}" STREQUAL "-1") # not a system directory
set(CMAKE_INSTALL_RPATH "${CMAKE_INSTALL_RPATH}:${CMAKE_INSTALL_PREFIX}/lib")
endif("${isSystemDir}" STREQUAL "-1")

# not sure if this is useful any more, but what the heck.
foreach(lib ${Boost_LIBRARY_DIRS})
set(CMAKE_INSTALL_RPATH "${CMAKE_INSTALL_RPATH}:${lib}")
endforeach()

## Not setting runpath because having MPI libs in LD_LIBRARY_PATH was messing up VampirTrace's ability to find its own libs. Maybe there's another way to fix this, but it just seems more robust (for now) to not allow LD_LIBRARY_PATH to affect our libs (after configure uses it to find them).
# set runpath, too
# if(NOT APPLE)
# set(CMAKE_EXE_LINKER_FLAGS "-Wl,--enable-new-dtags")
# endif()


##### </RUNPATH> ########

else()
set(CMAKE_INSTALL_RPATH "")
endif()

# RPATH
# not sure if this is useful any more, but what the heck.
foreach(lib ${Boost_LIBRARY_DIRS})
set(CMAKE_INSTALL_RPATH "${lib}:${CMAKE_INSTALL_RPATH}")
endforeach()



Expand Down Expand Up @@ -491,51 +504,68 @@ endif()
####################################
# Build lists of library names and paths

# static libs
foreach(lib ${CMAKE_BINARY_DIR}/system/libGrappa.a ${GRAPPA_STATIC_LIBS})
# fixup library lists in case things got added to the wrong one
foreach(lib ${GRAPPA_DYNAMIC_LIBS})
get_filename_component(ext "${lib}" EXT)
if("${ext}" STREQUAL ".a")
list(APPEND GRAPPA_STATIC_LIBS "${lib}")
else()
list(APPEND GRAPPA_CLEANED_DYNAMIC_LIBS "${lib}")
endif()
endforeach()
foreach(lib ${GRAPPA_STATIC_LIBS})
get_filename_component(ext "${lib}" EXT)
if("${ext}" STREQUAL ".so")
message(WARNING "Adding to dynamic list: ${lib}")
list(APPEND GRAPPA_DYNAMIC_LIBS "${lib}")
else()
list(APPEND GRAPPA_CLEANED_STATIC_LIBS "${lib}")
endif()
endforeach()

# generate list of dynamic libs
foreach(lib ${GRAPPA_CLEANED_DYNAMIC_LIBS})

# add directory to search paths
get_filename_component(dir ${lib} PATH) # get directory for libs
list(APPEND GRAPPA_LIB_PATHS ${dir})
string(FIND "${dir}" "${CMAKE_BINARY_DIR}" STATIC_IN_BUILD_DIR) # -1 if not found, 0 if found
string(FIND "${dir}" "${THIRD_PARTY_ROOT}" STATIC_IN_TPR_DIR) # -1 if not found, 0 if found
if(NOT STATIC_IN_BUILD_DIR EQUAL 0 AND NOT STATIC_IN_TPR_DIR EQUAL 0)
list(APPEND GRAPPA_LIB_PATHS_INSTALLED ${dir})
get_filename_component(dir "${lib}" PATH)
list(APPEND GRAPPA_LIB_PATHS "${dir}")
string(FIND "${dir}" "${CMAKE_BINARY_DIR}" DYNAMIC_IN_BUILD_DIR) # -1 if not found, 0 if found
string(FIND "${dir}" "${THIRD_PARTY_ROOT}" DYNAMIC_IN_TPR_DIR) # -1 if not found, 0 if found
if(NOT DYNAMIC_IN_BUILD_DIR EQUAL 0 AND NOT DYNAMIC_IN_TPR_DIR EQUAL 0)
list(APPEND GRAPPA_LIB_PATHS_INSTALLED "${dir}")
endif()

# add library to link list
get_filename_component(name ${lib} NAME_WE) # remove any extension
string(SUBSTRING ${name} 3 -1 name) # chop off "lib"
list(APPEND GRAPPA_STATIC_LIB_NAMES ${name})
get_filename_component(name "${lib}" NAME_WE)
string(SUBSTRING "${name}" 3 -1 name)
list(APPEND GRAPPA_DYNAMIC_LIB_NAMES "${name}")

# check for reasonable extension
get_filename_component(ext ${lib} EXT)
if(NOT ${ext} STREQUAL ".a")
message(WARNING "Static library found didn't have .a extension: ${lib}")
get_filename_component(ext "${lib}" EXT)
if(NOT "${ext}" STREQUAL ".so")
message(WARNING "Dynamic library found didn't have .so extension: ${lib}")
endif()
endforeach()

# dynamic libs
foreach(lib ${GRAPPA_DYNAMIC_LIBS})
# static libs
foreach(lib "${CMAKE_BINARY_DIR}/system/libGrappa.a" ${GRAPPA_CLEANED_STATIC_LIBS})

# add directory to search paths
get_filename_component(dir ${lib} PATH)
list(APPEND GRAPPA_LIB_PATHS ${dir})
string(FIND "${dir}" "${CMAKE_BINARY_DIR}" DYNAMIC_IN_BUILD_DIR) # -1 if not found, 0 if found
string(FIND "${dir}" "${THIRD_PARTY_ROOT}" DYNAMIC_IN_TPR_DIR) # -1 if not found, 0 if found
if(NOT DYNAMIC_IN_BUILD_DIR EQUAL 0 AND NOT DYNAMIC_IN_TPR_DIR EQUAL 0)
list(APPEND GRAPPA_LIB_PATHS_INSTALLED ${dir})
get_filename_component(dir "${lib}" PATH) # get directory for libs
list(APPEND GRAPPA_LIB_PATHS "${dir}")
string(FIND "${dir}" "${CMAKE_BINARY_DIR}" STATIC_IN_BUILD_DIR) # -1 if not found, 0 if found
string(FIND "${dir}" "${THIRD_PARTY_ROOT}" STATIC_IN_TPR_DIR) # -1 if not found, 0 if found
if(NOT STATIC_IN_BUILD_DIR EQUAL 0 AND NOT STATIC_IN_TPR_DIR EQUAL 0)
list(APPEND GRAPPA_LIB_PATHS_INSTALLED "${dir}")
endif()

# add library to link list
get_filename_component(name ${lib} NAME_WE)
string(SUBSTRING ${name} 3 -1 name)
list(APPEND GRAPPA_DYNAMIC_LIB_NAMES ${name})
# add library to link list (use full path for static linking)
list(APPEND GRAPPA_STATIC_LIB_NAMES "${lib}")

# check for reasonable extension
get_filename_component(ext ${lib} EXT)
if(NOT ${ext} STREQUAL ".so")
message(WARNING "Dynamic library found didn't have .so extension: ${lib}")
get_filename_component(ext "${lib}" EXT)
if(NOT "${ext}" STREQUAL ".a")
message(WARNING "Static library found didn't have .a extension: ${lib}")
endif()
endforeach()

Expand All @@ -553,16 +583,17 @@ list(REMOVE_DUPLICATES GRAPPA_LIB_PATHS_INSTALLED)
string(REPLACE ";" " \\\n\t${CMAKE_LIBRARY_PATH_FLAG}" GMAKE_LIB_PATHS_INSTALLED ";${GRAPPA_LIB_PATHS_INSTALLED}")

list(REMOVE_DUPLICATES GRAPPA_STATIC_LIB_NAMES)
string(REPLACE ";" " \\\n\t${CMAKE_LINK_LIBRARY_FLAG}" GMAKE_STATIC_LIBS ";${GRAPPA_STATIC_LIB_NAMES}")
string(REPLACE ";" " \\\n\t" GMAKE_STATIC_LIBS ";${GRAPPA_STATIC_LIB_NAMES}")

list(REMOVE_DUPLICATES GRAPPA_DYNAMIC_LIB_NAMES)
string(REPLACE ";" " \\\n\t${CMAKE_LINK_LIBRARY_FLAG}" GMAKE_DYNAMIC_LIBS ";${GRAPPA_DYNAMIC_LIB_NAMES}")

# fill in prototype file and place in build directory
configure_file(util/grappa.mk util/grappa.mk)
configure_file("util/grappa.mk" "util/grappa.mk")

#
# Installation
#
install(FILES ${CMAKE_BINARY_DIR}/util/grappa.mk DESTINATION "share/Grappa/")
install(FILES "${CMAKE_BINARY_DIR}/util/grappa.mk"
DESTINATION "share/Grappa/")

Loading