Manual Installation

This section will walk through the steps to creating a basic directory layout, cloning exawind-builder repository. In this example, we will create the exawind base directory within the user’s home directory. Modify this appropriately.

cd ${HOME}  # Change this to your preferred location

# Create the basic directory layout
mkdir -p exawind/{source,install,scripts}

# Clone exawind-builder repo
cd exawind
git clone https://github.com/exawind/exawind-builder.git

# Clone nalu-wind that we will use as an example later
cd ../source
git clone https://github.com/exawind/nalu-wind.git

If you are working on a system where the dependencies are already installed in a shared project location, then you can skip the next location and go to Generating Build Scripts.

Setting up dependencies

This section details basic steps to install all dependencies from scratch and have a fully independent installation of the ExaWind software ecosystem. This is a one-time setup step.

Mac OS X users will need to setup Homebrew as described in Initial Homebrew Setup for Mac OS-X Users before proceeding.

Install dependencies via spack (all systems)

Setup ExaWind directory structure as described in Exawind directory structure.

  1. Clone the spack repository

    cd ${HOME}/exawind
git clone https://github.com/spack/spack.git

# Activate spack (for the remainder of the steps)
source ./spack/share/spack/setup-env.sh

  2. Copy package specifications for Spack. The file packages.yaml instructs Spack to use the installed compilers and MPI packages thereby cutting down on build time. It also pins other packages to specific versions so that the build is consistent with other machines.

    cd ${HOME}/exawind/exawind-builder/etc/spack/osx
cp packages.yaml ${HOME}/.spack/$(spack arch -p)/

    The above example shows the configuration of OSX. Choose other appropriate directory within spack_cfg. Spack configs for other systems can be adapted from the build-test repository.

    Users can also copy compilers.yaml if desired to override default compilers detected by spack.

    Note

    For automatic updates, users can symlink the packages.yaml file within the spack configuration directory to the version in exawind-builder

    ln -s ${HOME}/exawind/exawind-builder/etc/spack/${SYSTEM}/packages.yaml ${HOME}/.spack/$(spack arch -p)/

  3. Setup compilers to be used by spack. As with packages.yaml, it is recommended that the users use the compiler configuration provided with exawind-builder.

    cp compilers.yaml ${HOME}/.spack/$(spack arch -p)/

    For more flexibility, users can use spack to determine the compilers available on their system.

    spack compiler find

    The command will detect all available compiler on users environment and create a compilers.yaml in the $HOME/.spack/$(spack arch -p).

    Note

    If you have multiple compilers.yaml in several locations, make sure that the specs are not conflicting. Also check packages.yaml to make sure that the compilers are listed in the preferred order for spack to pick up the right compiler.

  4. Instruct spack to track packages installed via Homebrew. Note that on most systems the following commands will run very quickly and will not attempt to download and build packages.

    spack install cmake
spack install mpi
spack install m4
spack install zlib
spack install libxml2
spack install boost

  5. Install remaining dependencies via Spack. The following steps will download, configure, and compile packages.

    # These dependencies must be installed (mandatory)
spack install superlu
spack install hdf5
spack install netcdf
spack install yaml-cpp

# These are optional
spack install openfast
spack install hypre
spack install tioga

    It is recommended that you build/install Trilinos using the build scripts described in Using exawind-builder to build software section. The optional dependencies could be installed via that method also.

  6. Generate build scripts as described in Generating Build Scripts section. On OS X, use -s spack for the system when generating the build scripts. For Cori and SummitDev, use the appropriate system which will initialize the compiler and MPI modules first and then activate Spack in the background. You will need to configure at least SPACK_ROOT if it was not installed in the default location suggested in the directory layout at the beginning of this section.

Upon successful installation, executing spack find at the command line should show you the following packages (on Mac OSX)

$ spack find
==> 12 installed packages.
-- darwin-sierra-x86_64 / clang@9.0.0-apple ---------------------
boost@1.67.0  libxml2@2.2     netlib-lapack@3.8.0    superlu@4.3
cmake@3.12.0  m4@1.4.6        openmpi@3.1.1          yaml-cpp@develop
hdf5@1.10.1   netcdf@4.4.1.1  parallel-netcdf@1.8.0  zlib@1.2.8

Generate builder configuration

Create your specific configuration in $HOME/exawind/exawind-config.sh. A sample file is shown below

### Example exawind-config.sh file
#
# Note: these variables can be overridden through the script in build directory
#

# Specify path to your own Spack install (if not in default location)
export SPACK_ROOT=${HOME}/spack

# Track trilinos builds by date
# export TRILINOS_INSTALL_DIR=${EXAWIND_INSTALL_DIR}/trilinos-$(date "+%Y-%m-%d")

### Specify custom builds for certain packages. The following are only
### necessary if you didn't install these packages via spack, but instead are
### using your own development versions.
export TRILINOS_ROOT_DIR=${EXAWIND_INSTALL_DIR}/trilinos
export TIOGA_ROOT_DIR=${EXAWIND_INSTALL_DIR}/tioga
export HYPRE_ROOT_DIR=${EXAWIND_INSTALL_DIR}/hypre
export OPENFAST_ROOT_DIR=${EXAWIND_INSTALL_DIR}/openfast

# Turn on/off certain TPLs and options
ENABLE_OPENMP=OFF
ENABLE_TIOGA=OFF
ENABLE_OPENFAST=OFF
ENABLE_HYPRE=OFF

See Reference for more details. Note that the default path for Spack install is $EXAWIND_PROJECT_DIR/spack.

Generating Build Scripts

exawind-builder provides a new-script.sh command to generate build scripts for combination of system, project, and compiler. The basic usage is shown below

bash$ ./new-script.sh -h
new-script.sh [options] [output_file]

Options:
  -h             - Show help message and exit
  -p <project>   - Select project (nalu-wind, openfast, etc)
  -s <system>    - Select system profile (spack, peregrine, cori, etc.)
  -c <compiler>  - Select compiler type (gcc, intel, clang)

Argument:
  output_file    - Name of the build script (default: '$project-$compiler.sh')

So if the user desires to generate a build script for Trilinos on the NERSC Cori system using the Intel compiler, they would execute the following at the command line

# Switch to scripts directory
cd ${HOME}/exawind/scripts

# Declare project directory variable (default is parent directory of exawind-builder)
export EXAWIND_PROJECT_DIR=${HOME}/exawind

# Create the new script
../exawind-builder/new-script.sh -s cori -c intel -p trilinos

# Create a script with a different name
../exawind-builder/new-script.sh -s cori -c intel -p trilinos trilinos-haswell.sh

Creating runtime environment script

exawind-builder provides a create-env.sh command to generate a source-able script that can be used within job submission scripts as well as to recreate the environment used to build the code in interactive shells.

create-env.sh [options] [output_file_name]

By default it will create a file called exawind-env-$COMPILER.sh

Options:
  -h             - Show help message and exit
  -s <system>    - Select system profile (spack, cori, summitdev, etc.)
  -c <compiler>  - Select compiler type (gcc, clang, intel, etc.)

Sample usage shown below

# Create a new environment file
cd ${HOME}/exawind/scripts

../exawind-builder/create-env.sh -s eagle -c gcc

# Source the script within interactive shells
source ./exawind-env-gcc.sh

# Load additional modules
exawind_load_deps hdf5 netcdf

It is recommended that the user use exawind_load_deps() instead of spack load or module load as this has several advantages: exawind-builder will automatically use spack/module to load depending on the system you are on, so you can use one command across different systems; this command will respect EXAWIND_MODMAP and load the appropriate module that you have configured instead of the defaults on the system; it will configure CUDA based on ENABLE_CUDA.

Configuring exawind-builder to use Ninja

Ninja is a build system that is an alternative to make. It provides several features of make but is considerably faster when building code. The speedup is particularly evident when compiling Trilinos. Since codes used in ExaWind project contain Fortran files, it requires a special fork of Ninja (maintained by Kitware). exawind-builder provides a script get-ninja.sh to fetch and configure Ninja for builds.

# Get Ninja
cd ${HOME}/exawind
./exawind-builder/utils/get-ninja.sh

Note

You will need to execute cmake_full to force CMake to recreate build files using Ninja if they were previously configured to use Makefiles.

Compiling Nalu-Wind

At this point you have manually recreated all the steps performed by the bootstrap process. Please follow Compiling Software to build Trilinos and Nalu-Wind