Run MUSC with EMS

These instructions require the use of dev-CY46h1.

If you find any issues with any of the instructions or scripts, feel free to notify Emily Gleeson (emily.gleesonATmet.ie) and Eoin Whelan (eoin.whelanATmet.ie)

In this section a description of how to install and run MUSC using EMS is provided. This is based on compilation and execution in a Ubuntu 20.04 container (tested using Apptainer on the ECMWF Atos HPC) and use of the EMS system to execute MUSC and convert the output to NetCDF. EMS is primarily developed by Romain Roehrig (Météo France) https://github.com/romainroehrig/EMS.

Start your container

1. Log in to hpc-login on the Atos
2. Load the Apptainer module and start the Ubuntu 20.04 container:

module load apptainer
/home/dui/musc_ubuntu.sif

Further details concerning Apptainer on the Atos are available here

Compile the code

I (Eoin) have not had time to sort out compilation using CMake but the following instructions provide a minimalist approach to compile the code using makeup. These instructions depend on you having a clone or copy of Harmonie (dev-46h1) in your $PERM directory on Atos where GHUSER is your Github username.

cd $PERM
GHUSER=your_github_username
git clone git@github.com:$GHUSER/Harmonie.git harmonie_git/$GHUSER/dev-CY46h1 -b dev-CY46h1

(Ensure that your fork is up-to-date before doing this or else take the code from the main
CY46 repo. Note that to run GABLS1 you need to use two updated surfex subroutines `tszo.F90`
and `read_pgd_tsz0_parn.F90`. You can copy these files from util/musc/patches as follows:

cp $PERM/harmonie_git/$GHUSER/dev-CY46h1/util/musc/patches/tsz0.F90 $PERM/harmonie_git/$GHUSER/dev-CY46h1/src/surfex/SURFEX/tsz0.F90
cp $PERM/harmonie_git/$GHUSER/dev-CY46h1/util/musc/patches/read_pgd_tsz0_parn.F90 $PERM/harmonie_git/$GHUSER/dev-CY46h1/src/surfex/SURFEX/read_pgd_tsz0_parn.F90
or by using git apply gabls.patch).

Compile the code in your code checkout/copy:

cd $PERM/harmonie_git/$GHUSER/dev-CY46h1
EXP=$(basename $(git symbolic-ref --short HEAD))
. config-sh/config.ubuntu20_nompi
util/makeup/build -n 4 config.${HARMONIE_CONFIG}
mkdir ${EXP}
mv makeup.${HARMONIE_CONFIG} bin configlog.1 makelog.1 ${EXP}/

Install EMS

The following instructions provide details on how to download a HIRLAM version of EMS and install locally in your own account:

cd $HOME
git clone git@github.com:ewhelan/EMS.git -b hirlam EMS_git/EMS
cd EMS_git/EMS/
mkdir build
cd build/
export EMS_DIR=$HOME/metapp/ems
cmake .. -DCMAKE_INSTALL_PREFIX=$EMS_DIR && make && ctest && make install
export PATH=${EMS_DIR}/bin:$PATH
export PYTHONPATH=${EMS_DIR}

nam[nn]['XUNIF_CLAY'] = ['1.']
nam[nn]['XUNIF_SAND'] = ['0.']
nam[nn]['XUNIF_RUNOFFB'] = ['0.5']

nam[nn]['XHUG_SURF'] = ['-10.']
nam[nn]['XHUG_ROOT'] = ['-10.']
nam[nn]['XHUG_DEEP'] = ['-10.']

Run EMS

Here are some instructions on how to use EMS to execute idealised cases

export EMS_DIR=$HOME/metapp/ems
export PATH=${EMS_DIR}/bin:$PATH
export PYTHONPATH=${EMS_DIR}
mkdir $HOME/ems_exec
cd $HOME/ems_exec
cp ${EMS_DIR}/share/config/config_46h1_HARMAROME_DEV.py .
### edit his file to point to your binaries
ems_list_cases.py
export PYTHONPATH=$(pwd):$PYTHONPATH
# MUSC.py -config config_46h1_HARMAROME_DEV.py -case $CASE  -subcase $SUBCASE
MUSC.py -config config_46h1_HARMAROME_DEV.py -case ARMCU  -subcase REF

Output and log file can be found in $HOME/ems_exec/simulations/46t1/46h1_HARMAROME_DEV/${CASE}/${SUBCASE}

EMS Cases

The table below lists the cases available in EMS and results of early tests.

Analysing results using Atlas

Still in the Apptainer container

cd $HOME
git clone git@github.com:ewhelan/SCM-atlas.git -b hirlam SCM-atlas_git/ewhelan/hirlam
export PATH=$HOME/SCM-atlas_git/ewhelan/hirlam/apptools:$PATH
export PYTHONPATH=$HOME/SCM-atlas_git/ewhelan/hirlam:$PYTHONPATH
export ATLAS_CONFIG=""
mkdir -p $HOME/Atlas
cd $HOME/Atlas
mkdir config
cp $HOME/SCM-atlas_git/ewhelan/hirlam/examples/config/config_HARM.py config/
### edit config/config_HARM.py
run_atlas1d.py -config config/config_HARM.py