Build with CMake

Background

CMake is a build system generator supporting multiple build systems and programming languages, specifically Fortran is a first-class citizen there, allowing, for example, out-of-the-box handling of the inter-module dependencies. A build system generator there means that description of the build procedure written in the CMake-script language is used by the cmake tool to generate the actual build system, for example using Unix Makefiles or Ninja generator. Thus, all modifications should be performed on the CMake-script level and not within the generated build system as these changes will be overwritten when re-running cmake at some point.

Why providing yet another alternative for building HARMONIE-AROME? Well, makeup does a very good job building the system, however it's an in-house solution which has a number of limitations:

• makeup is an in-house build system, so there are components that require more maintenance compared to a standardized build tool
• makeup uses a considerable number of sequential steps, which increase the total build time
• the configure step takes quite some time, although in some cases it can be skipped, but users have to remember when they must re-run configure and this dependency is not enforced by makeup
• not all the dependencies are tracked by makeup, for example updating configure files does not trigger a re-build

In an attempt to fix these limitation of makeup, CMake was chosen as an alternative. CMake has a mature Fortran support and improves upon some shortcomings of makeup with little effort (well, it obviously has its own fair share of quirks, but that's a different story...). Additionally, using CMake allows us to enforce usage requirements and dependencies between different components of HARMONIE-AROME, for example, it's a good idea to ensure that SURFEX routines do not directly call cloud microphysics functions. Currently makeup does not enforce these boundaries and this task is left to the developers who implement the new code. Of course, something like this can also be implemented with makeup, but it would require considerable development efforts.

Getting started with CMake

Selecting the CMake-based build system when installing HARMONIE-AROME

If all the config files are available, building HARMONIE-AROME with CMake should be as simple as setting the BUILD_WITH variable when invoking Harmonie:

config-sh/Harmonie install BUILD_WITH=cmake

or alternatively, setting the desired option in ecf/config_exp.h.

Building HARMONIE-AROME with CMake from the command line

Sometimes calling Harmonie install is not the best choice and one might want to compile the code from the command line. In this case compilation of HARMONIE-AROME with CMake consists of three individual steps:

1. compiling the auxiliary libraries (gribex and such)
2. compiling the main code of HARMONIE-AROME
3. optionally, compile some additional tools (for example, gl)

1. Compiling the auxiliary libraries

This step is rather straightforward, assuming that HARMONIE-AROME code is located under the path stored in the HM_LIB environment variable one can adapt the following snippet to compile all the required libraries:

CMAKE_FLAGS="-DCONFIG_FILE=<path to your JSON config>"
INSTALL_DIR="<directory where the auxiliary libraries should be installed>"

AUX_LIBS='bufr_405 gribex_370 rgb_001 dummies_006/mpidummy'
for project in $AUX_LIBS; do
  echo "Compiling $project"
  current_project_dir=$HM_LIB/util/auxlibs/$project
  current_build_dir="build-`echo $project | sed 's|/|-|g'`"

  mkdir -p $current_build_dir && cd $current_build_dir

  # CMake build type can be changed to Debug, if needed
  cmake $current_project_dir -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=$INSTALL_DIR $CMAKE_FLAGS
  # Here -j tells CMake how many parallel compilation processes to use
  cmake --build . --target install -j16

  cd ..
done

If a specific config file is not there, you can try your luck with using generic config files provided for different compiler types. To do so, just drop the -DCONFIG_FILE from the list of CMake command line arguments and CMake will try to load a suitable configuration file, if available.

2. Compiling the main code of HARMONIE-AROME

Following the procedure described in the previous step, one can use a similar approach to compile the main code (here, one of the generic configuration files is used, of course it can be replaced with a different one or dropped but it should be the same config file which was used to compile auxiliary libraries):

mkdir build && cd build
# Configure and generate the build system
cmake $HM_LIB/src \
  -G Ninja # Use Ninja to build HARMONIE-AROME, drop to build with Makefiles
  -DCMAKE_BUILD_TYPE=Release \
  -DCONFIG_FILE=$HM_LIB/util/util/cmake/config.GNU.cmake \
  -Dbufr_DIR=$INSTALL_DIR/lib/cmake/bufr \
  -Dgribex_DIR=$INSTALL_DIR/lib/cmake/gribex \
  -Drgb_DIR=$INSTALL_DIR/lib/cmake/rgb \
  -DCMAKE_INSTALL_PREFIX=$INSTALL_DIR

# Build and install HARMONIE-AROME
cmake --build . --target install -j16

3. Compiling the tools

The approach is the same as with the main code, however, you might want to add -Dharmonie_DIR=$INSTALL_DIR/lib/cmake/harmonie if the tool in question needs HARMONIE-AROME libraries for compilation.

Configuration files

Configuration files, similarly to makeup, are used to provide compilation flags, define external libraries to use when compiling the code et cetera. Thus, having a correct configuration file is one of the key elements of successful building HARMONIE-AROME. The CMake-based build system of HARMONIE-AROME uses configuration files written in JSON format. The JSON configuration files use the following naming convention: config.PLATFORM.COMPILER.json where PLATFORM is the name of your PC/HPC/OS and COMPILER is the compilier identification as specified by CMake in https://cmake.org/cmake/help/latest/variable/CMAKELANGCOMPILER_ID.html. JSON was chosen to make these files more declarative and, hopefully, easier to maintain and modify than plain CMake-script-based files would be.

The main config file, which is used to build auxiliary libraries and the main HARMONIE-AROME code should be placed under util/cmake/config directory. This file has a following top-level structure:

{
  "build_tools":[],
  "dependencies":[],
  "programs":[],
  "configure":{},
  "compile":[],
  "compile_single":[],
  "compile_double":[],
  "custom_compile":{},
  "link":[]
}

there all the sections except configure, custom_compile and link are mandatory. In the following a detailed description of all the config file section is provided.

The build_tools section

This section lists the external tools required for compiling HARMONIE-AROME, excluding compilers. Currently, this section should always contain the two following entries: FLEX and BISON, but in future this list might be extended. So, currently this section is always defined as:

"build_tools":["BISON", "FLEX"]

The dependencies section

This section provides a list of external (external here means "not found within the src directory of HARMONIE-AROME", so, for example, gribex is also an external library for CMake build) libraries required to compile and link HARMONIE-AROME code. Since finding a correct library can be a tricky task, this section allows a number of options for specifying external dependencies:

• You can completely rely on CMake and delegate it all the work for finding a dependency. In this case, a dependency is added as a simple string to the dependencies section, for example:

  "dependencies":["OpenMP", "LAPACK"]

  This option is for packages like OpenMP which do not involve finding libraries located in unusual places as often happens when using environment modules.

• You can still rely on CMake to find the package, but provide a bit of detail on how to find it. In this case a dependency is added as a JSON object of the following form (using the NetCDF library as an example):

  {
    "pkg":"NetCDF",
    "use_cmake_config":false,
    "components":["C","Fortran"],
    "hints":["$ENV{NETCDF_DIR}","$ENV{NETCDF_F_DIR}"],
    "cmake":{"NETCDF_USE_DEFAULT_PATHS":true}
  }

  There the use_cmake_config field tells CMake which mechanism it should use to find the library in question. When use_cmake_config is set to true CMake will look for CMake configuration files installed with the library, which is a recommended option in modern CMake. Even though it's a recommended by the CMake authors option, not all the libraries provide CMake configuration files so just setting use_cmake_config to true does not work all the time (at least it works for the auxiliary libraries compiled with CMake). You might want to provide a -D<package name>_DIR=<path to CMake config files> as an argument to the cmake command when configuring the build if CMake fails to find a package.

  Another alternative is setting use_cmake_config to false, then CMake will try to find the required dependency using the hand-written scripts provided by the authors of CMake (or found under the util/cmake directory of HARMONIE-AROME). These scripts usually do quite some work trying to find a dependency and sometimes fail even if library is there, for example when it's located in a very unusual place or has an unexpected pkg-config name.

  When using "use_cmake_config":false one may add a components list, if only a language-specific version of the dependency is wanted. For example, having:

  {"pkg":"NetCDF", "use_cmake_config":false, "components":["C"]}

  CMake would not try to find the Fortran version of NetCDF library, which can be useful sometimes. Use this option of defining external dependencies for such libraries as MPI, which can have multiple vendors and subtle differences between libraries provided (for example CMake should be able to figure out the correct MPI libraries for both MPICH and Open-MPI).

  The hints list tells CMake which directories it should check when looking for a library.

  Note

  Elements of the hints list are simply added to the <PackageName>_ROOT CMake variable. If CMake's find_package(<PackageName>) does not use this variable providing hints would have no effect.

  Finally, the cmake section provides a key-value set of elements, which will be converted to corresponding CMake variables set before calling find_package(<PackageName>). Thus, it can be used to control the behaviour of find_package.

  Note

  Variables set in the cmake section are local to the current package and do not modify the global scope.

• When nothing of the above works, you can provide all the flags manually. To do so, use the following form for a dependency entry:

  {
    "pkg":"HDF5",
    "raw_lib":{
      "include":"$ENV{HDF5_DIR}/include",
      "lib_directory":"$ENV{HDF5_DIR}/lib",
      "lib":["-lhdf5hl_fortran", "-lhdf5_fortran", "-lhdf5_hl", "-lhdf5"]
    }
  }

  where the raw_lib component provides all the needed include and link directories as well as the link libraries. If some some fields of the raw_lib object are unneeded they can be set to null:

  {"pkg":"rt", "raw_lib":{"include":null, "lib_directory":null, "lib":"-lrt"}}

  Note that all the members of the raw_lib object can be defined as lists:

  {
    "pkg":"HDF5",
    "raw_lib":{
      "include":["$ENV{HDF5_DIR}/include","$ENV{HDF5_DIR}/include_fortran"],
      "lib_directory":["$ENV{HDF5_DIR}/lib","$ENV{HDF5_DIR}/lib64"],
      "lib":["-lhdf5hl_fortran", "-lhdf5_fortran", "-lhdf5_hl", "-lhdf5"]
    }
  }

  When providing the required libraries in the lib section one can skip the -l prefix, thus having "lib":"-lrt" and "lib":"rt" would have the same effect.

• Sometimes it can be useful to define a dummy library in CMake without actually looking for the library files, for example when compiling a tool which uses only a subset of HARMONIE-AROME libraries. When loading HARMONIE-AROME as a CMake package all the targets associated with external dependencies should be present, but some of these dependencies might be not needed for successful linking (or these are added implicitly by the programming environment and adding them for the second time in CMake won't make any difference). In this case you can use the following:

  {"pkg":"gribex",  "dummy":true}

The programs section

This section provides a list of HARMONIE-AROME programs to build (excluding MASTERODB which is always built by the CMake build system), for example:

"programs":["BATOR", "oulan", "ioassign", "LSMIX"]

CMake will try to find the corresponding Fortran source files and will complain if unable to do so. Currently it is not possible to explicitly tell CMake via JSON config which program should be compiled from which source file. If CMake is unable to figure out how to compile a program the CMake-code should be altered to tell it how to do so.

The configure section

This section provides various configure-time flags controlling the build system or selecting features. Currently in the main HARMONIE-AROME config file this section is defined as follows:

"configure":{
    "use_flexfix":true
  , "precision":"double"
},

There the use_flexfix option controls the usage of the flexfix wrapper, set it to true to use the flexfix wrapper when generating lexers for the Blacklist and ODB compilers. Having use_flexfix as false results in using the flex tool directly. The precision option controls the floating point precision of the build, with possible values of double and single. This is a mandatory option, removing it would result in a CMake fatal error at the configure time.

The configure file for gl has the following options in the configure section:

"configure":{
    "use_aladin":true
  , "use_netcdf":true
  , "check_preferlocalconcepts_bug":true
}

Set use_aladin to true to compile with FA support (requires HARMONIE-AROME libraries). Set use_netcdf to true to enable NetCDF support in gl. Set check_preferlocalconcepts_bug to true to perform a configure-time auto-detection test checking whether the supplied eccodes version is affected by the preferLocalConcepts bug. This test can be skipped, although in such a case corresponding CPP definitions should be manually added to the config file if a 'bad' eccodes version is used.

Adding a new configure option

There's no predefined list of configure section members of CMake JSON config, any element found in this section will be available as CONFIG_<option name> from CMake code. For example having:

"configure":{
    "use_flexfix":true
  , "precision":"double"
  , "new_flag":true
},

would define the following variables available in CMake scripts (after call to the hm_get_json_configure_options function): CONFIG_USE_FLEXFIX, CONFIG_PRECISION and CONFIG_NEW_FLAG. How these newly introduced options are used when configuring the build is up to the developer.

The compile section

This section defines compiler flags and preprocessor definitions which should be used when compiling various components of HARMONIE-AROME. Generally this section should consists of a list of objects with the following structure where the first of these objects should always have the project name set to null:

{
    "project":null
  , "flags_fortran":{"any":[], "debug":[], "release":[]}
  , "flags_c":[]
  , "defs_fortran":[]
  , "defs_c":[]
  , "exclusive_defs":true
  , "exclusive_flags":true
},

Individual members of these objects are defined as follows:

• project is the name of the CMake target for which the provided setting should be applied. CMake matches this field against the names of its targets, which means that having, for example, "project":"surf" would apply the compilation flags for all CMake targets named surf or with names starting with surf-, as in surf-module. When CMake finds a suitable project-specific section it will append project-specific compilation flags to the global compilation flags for the project in question (if not explicitly asked to not use global flags). If there's no suitable section with project-specific options, only the global options will be used.

  Note

  All compilation flags and preprocessor definitions should be set via JSON config and not through the native CMake variables.

• flags_fortran define compiler flags to be used when compiling Fortran source files. Flags can be defined in three different ways: as a string (e.g., "flags_fortran":"-std=f2003"), as a list (e.g., "flags_fortran":["-std=f2003", "-Wall"]) or as an object. Providing the compiler flags as a JSON object allows for a fine-grained control applying different flags for different build configurations (the first two forms will apply the same flags for any build configuration). This object contains three subsections: any, debug and release. Compiler flags put under the any section will be used for any build type, while flags under debug and release will be used only when CMAKE_BUILD_TYPE is set to Debug or Release, respectively. Thus if having, for example, "flags_fortran":{"any":["-std=f2003"], "debug":["-O0"], "release":["-O2"]} Fortran sources will be compiled using -std=f2003 -O0 when CMAKE_BUILD_TYPE=Debug and -std=f2003 -O2 when CMAKE_BUILD_TYPE=Release.

  Note

  Technically, the compiler flags there are not limited to the debug and release configurations. CMake allows defining custom CMAKE_BUILD_TYPE and it will look for a correspondingly named element under the flags_fortran object. However, this feature is not used in the current version of CMake-based build system for HARMONIE-AROME

  Note

  Compiler flags there can be provided as a list (e.g., ["-Wall", "-Wextra"]) or as a plain string when there is only a single flag (e.g., "flags_fortran":"-O2") but never as a space-separated string with multiple compiler flags. Trying to do so will result in an error.

• flags_c same as flags_fortran but for compiling C code.

• defs_fortran defined in the same way as flags_fortran (allowing the same three forms: a string, a list or an object) but contains preprocessor definitions.

  Note

  Preprocessor definitions should be added without the -D prefix, for example: "defs_fortran":{"debug":"PRINT_MORE"} and not as defs_fortran":{"debug":"-DPRINT_MORE"}.

• defs_c same as defs_fortran but for the C code.

• exclusive_defs and exclusive_flags flags. In some situations it can be useful to ignore the global compilation flags defined by "project":null and use only project-specific flags or preprocessor definitions. To do so, add exclusive_defs and/or exclusive_flags to the project-specific element of the compile section, for example having:

    {
        "project":null
      , "flags_fortran":"-Wall"
    },
    {
        "project":"gribex"
      , "exclusive_defs":true
      , "exclusive_flags":true
      , "flags_fortran":"-O0"
    },
    {
        "project":"surfex"
      , "flags_fortran":"-O0"
    }

  will result in having gribex to be always compiled with just -O0 without using any other flags or preprocessor definitions, but surfex will be compiled with -Wall -O0.

The compile_single and compile_double sections

These sections repeat the structure of the compile section. Compiler flags defined by these sections are appended to the flags defined in the compile section of the config based on the value of the value of the precision flag from the configure section.

The custom_compile section

Sometimes it might be desirable to add some specific compile flags for a single source file. This can be achieved by using the custom_compile section of JSON config as follows:

"custom_compile":{
  "sufpf.F90":["-O0", "-Wextra"]
},

The link section

This section provides a list of linker flags to be used when linking HARMONIE-AROME executables, for example:

"link":["LINKER:-export-dynamic,--as-needed", "-rdynamic"]

Using environment variables in JSON configuration files

Current version of CMake JSON config implemented in HARMONIE-AROME allows using environment variables as $ENV{VARIABLE} in the following contexts:

• string-valued members of the configure section, e.g., "configure":{"precision":"$ENV{VARIABLE}"}
• any elements of JSON config which allow a list of strings as one of the possible values, for example:
  • "link":["LINKER:-export-dynamic,--as-needed", "$ENV{VARIABLE}"] works since the link section expects a list of strings
  • "compile_double":[{"project":null, "defs_fortran":"$ENV{VARIABLE}"}] also works because defs_fortran allows a list of strings as an option
  • {"pkg":"$ENV{VARIABLE}","use_cmake_config":true} does not work because pkg is expected to provide a single string

Note on the structure of the JSON configuration files

The structure of JSON-based configuration files is not automagically understood by CMake, there is some boilerplate code to be added to CMake scripts. Thus, the described structure of the CMake config file fully applies only to the main build. Other components of the system try to use the same structure of the config file, but some sections might be not handled correctly yet (for example, if a project does not use configure-time flags adding a configure section without modifying the CMake code would not have any effect and corresponding CMake variable won't be added).

Note on the auto-double flags

The compiler flags defining the preferred precision of the floating point variables in CMake build for HARMONIE-AROME are provided on the per-compiler and not per-config level. Thus, for each compiler type recognized by CMake a file named as FortranCompilerFlags.<compiler type>.cmake should be added to the util/cmake directory. This file should define the following CMake variables Fortran_DEFAULT_FLOAT_32, Fortran_DEFAULT_FLOAT_64, Fortran_DEFAULT_INT_32, Fortran_DEFAULT_INT_64 (some of them may be empty if a compiler does not provide corresponding flags). For example for the GNU compilers FortranCompilerFlags.GNU.cmake is defined as:

set(Fortran_DEFAULT_FLOAT_32 "")
set(Fortran_DEFAULT_FLOAT_64 "-fdefault-double-8 -fdefault-real-8")

set(Fortran_DEFAULT_INT_32   "")
set(Fortran_DEFAULT_INT_64   "-fdefault-integer-8")

When running cmake configure, and depending on the build precision, a subset of these flags is added to the CMAKE_Fortran_FLAGS variable thus affecting all the Fortran targets. Currently, DEFAULT_INT variables are not used in CMake build, but are provided for consistency.

Note on generating different build systems with CMake

CMake is a build system generator and it can create different native build systems from the same CMakeLists.txt. The full list of supported generators is available in the CMake documentation, however in practice when building HARMONIE-AROME on a Linux machine (or on a UNIX-like one in general) there are two options: the Unix Makefiles generator and the Ninja generator:

• Unix Makefiles generator produces a build system based on the standard makefiles and does not use "exotic" tools. This is a default generator for CMake running on Linux and it usually works pretty well. However, when building with Unix Makefiles, CMake relies on its own Fortran parsers to scan the source tree and determine the build dependencies. Thus, in some rare cases of heavy CPP usage in Fortran code CMake can get inter-module dependencies wrong. The Unix Makefiles build is not parallel by default but it can be controlled, as with any conventional makefile-based build, by passing the desired -j flags to make. Additionally, when invoking the build via cmake --build command, a -j (or --parallel) flag can be used for setting the number of parallel jobs in a build-system-agnostic way, see CMake documentation.

• Ninja is a modern alternative to Make. Ninja is built with focus on speed and Ninja build is parallel by default, however, unlike Make, the build files for Ninja are very cumbersome to hand-write and they are usually machine-generated. When building Fortran code with CMake Ninja generator, an explicit preprocessing step is added, thus the inter-module dependencies should be always correct (or at least these corner cases where Unix Makefiles struggles to get correct dependencies are handled correctly by Ninja). In some cases using Ninja generator can reduce the build time due to better parallelization of the build, however since Ninja has a separate preprocessing step, it generates more output and, if the file system is a bottleneck, Ninja build can be slower than Unix Makefiles build. Using the Ninja generator in CMake requires the ninja tool to be available in the $PATH at the configure time.

Practical considerations

When to re-run CMake configure in my experiment?

In principle, it should be enough to run CMake configure only once to generate the build system and after that any modification of the source code or configuration files should be detected by the build system triggering the required re-build steps. The only time, when CMake configure should be explicitly re-run is when you add a new source file to HARMONIE-AROME. The current implementation of the CMake build scans the file system looking for the source files to compile, so just putting a new file under, say, src/surfex/SURFEX/ and re-running the build isn't enough since this new file would be still unknown to the build system, thus the need of rerunning the configure step first.

I added some code and CMake build stopped working

Unlike makeup, CMake build for HARMONIE-AROME enforces inter-project boundaries and each project has an explicit list of its dependencies. For example, it is not possible to use modules from arpifs in surfex, but it is possible to use mse modules. If after a code modification CMake starts complaining about missing module files, then it means that this modification violates the project dependencies in the build. To fix this problem, please update your changeset to use only the available modules. If you believe that your modification is sound with respect to inter-project dependencies of HARMONIE-AROME and it's the CMake build which misses a dependency, please open a new GitHub issue explaining the problem.

Can I move/copy my build directory to another directory and re-use it?

No, it's generally a bad idea. CMake loves absolute paths and uses them in many parts of the generated build system, thus simply moving the build directory would break the build.

Something went wrong and CMake doesn't behave anymore, can I refresh the build without nuking the whole build directory?

You can try deleting just the CMakeCache.txt file from the build directory.

CMake picks a wrong compiler

Sometimes CMake selects a system default compiler instead of the compiler provided, for example, by loading a module. There are a few options available to force CMake to use a specific compiler, a straightforward one is to set the compiler via commonly-used environment variables (for example, export FC=ifort for a Fortran compiler). Another way, is to set the correct compilers in command-line arguments when configuring the CMake build (for example adding -DCMAKE_Fortran_COMPILER=ifort to the list of CMake arguments). CMake recognizes CMAKE_<LANG>_COMPILER passed from the command line where <LANG> can be Fortran, C or CXX.

Can I get more verbose output when compiling with CMake?

To get detailed information about individual steps and commands issued when compiling HARMONIE-AROME with CMake add -v to your build command:

cmake --build . --target install -v

Is there a way to visualise dependencies between individual targets of HARMONIE-AROME in CMake build?

Since all the inter-target dependencies are defined in CMake scripts it can be useful to have an option to produce a graphical overview of the dependency graph of HARMONIE-AROME without grepping all the CMakeLists.txt files. This can be achieved by adding the --graphviz=<output file name> to the list of CMake arguments, for example:

cmake $HM_LIB/src --graphviz=harmonie.dot

then the produced dependency graph can be visualized using the dot tool:

dot -Tx11 harmonie.dot

The full dependency graph may be very cluttered and take quite some time to render, so it might be a good idea to plot dependencies of a single target, for example:

dot -Tx11 harmonie.dot.surf-static

See the CMake documentation on graphviz for additional information about fine-tuning of the generated graphs.

I need more information about CMake, where do I find documentation?

CMake documentation portal is a great source of detailed information about the various aspects of the CMake build system.