MUSC

MUSC using the develop branch (CY46) in the git repository

If you find any issues with any of the instructions or scripts, feel free to notify Emily Gleeson (emily.gleesonATmet.ie) and Eoin Whelan (eoin.whelanATmet.ie)

Currently a "reference" test case, called musc_ref, works on ATOS, as well as the ARMCU cases (with and without SURFEX for both AROME and HARMONIE namelists) and the two microphysics-related cases (supercooled liquid) developed by Bjorg Jenny Engdahl in cycle 40.

Some instructions on how to use MUSC are included below. See here for some information on HARMONIE-AROME experiments using MUSC but note that the scripts have changed somewhat since that paper was written.

Set up MUSC

Get the code:

mkdir -p $SCRATCH/harmonie_releases/git/HCY46
cd $SCRATCH/harmonie_releases/git/HCY46
git clone git@github.com:Hirlam/Harmonie.git
cd Harmonie
git checkout dev-CY46h1 
# If you already have a clone of the code but want to update it to the latest version, 
use "git pull" rather than "git branch".

Set up a MUSC experiment using HARMONIE scripting

In this example the ECMWF.atos config file is used.

mkdir -p $HOME/hm_home/test_0001
cd $HOME/hm_home/test_0001
$SCRATCH/harmonie_releases/git/HCY46/Harmonie/config-sh/Harmonie setup -r $SCRATCH/harmonie_releases/git/HCY46/Harmonie/ -h ECMWF.atos

Compile your experiment (still in $HOME/hm_home/test_0001)

$SCRATCH/harmonie_releases/git/HCY46/Harmonie/config-sh/Harmonie install BUILD_WITH=cmake
# Note that for the ARMCU cases cmake needs FFT modifications (not yet committed by Yurii)

Some MUSC specific settings including copying over scripts and a check that Harmonie setup has been run

$SCRATCH/harmonie_releases/git/HCY46/Harmonie/util/musc/scr/musc_setup.sh -r $SCRATCH/harmonie_releases/git/HCY46/Harmonie/

Generate your namelist, unless you're using an idealised case with pre-defined namelists (so for ARMCU* you do not generate the namelists for example). If you wish to change the radiation scheme (RADSCHEME- RAYFM (IFS) or RAY (ACRANEB2)) or how you use aerosols (BDAER - cams or none), you need to edit ecf/configexp.h in your expt before running muscnamelist.sh. For using NRT aerosols, they need to be included in your input files already e.g. the MUSCIN* files should come from a 3D NRT aerosol expt.
```
cd $HOME/hm_home/test_0001
./musc_namelist.sh -h
./musc_namelist.sh -l <length of run> -i <the ID name on the generated namelist files e.g. DEF> 
                     -[N nudging - optional]
```

Get a copy of the input files

cd $SCRATCH
retrieve the input files from https://github.com/Hirlam/HarmonieMuscData

Note that if you need to do experiments with 2 patches etc, ensure you derive some MUSC input files yourself using 3D HARMONIE-AROME files run with 2 patches. MUSC*REFL65* input files have only 1 patch. Changing MUSC namelists won't enable 2 patch output from a MUSC run.

Run MUSC

`Musc_ref`

The reference test is a X-hr experiment (change CSTOP in musc_namelist.sh if you wish to change the run length) and produces Out *lfa files for each model time-step of the time period. ICM* files are produced at each hour.

Run your experiment

cd $HOME/hm_home/test_0001
./musc_run.sh -h
./musc_run.sh -d $SCRATCH/muscCY46InputData/musc_ref -n REFL65 -i DEF [ -e ECOCLIMAP_PATH]
# optional path for ECOCLIMAP data may be given. For musc_ref -i must be given as no 
# namelists are provided with this experiment and must be generated before musc_run.sh 
# is executed. For the idealised cases, if -i is not specified -i becomes the name of 
# the idealised case once the namelist files are copied to $HOME/hm_home/test_0001 e.g. 
# for armcu the namelist files become namelist_atm_armcu etc.

ARMCU

** Note that these will not work until we can compile with FFTW.

This is an idealized SCM test case, the "Sixth GCSS WG-1 case (ARM—Atmospheric Radiation Measurement)", focussing on the diurnal cycle of cumulus clouds over land (Brown et al, 2002, Lenderink et al, 2004. The input files and namelist settings have been taken from /src/validation/mitraille/namelist/L1ARO but the atmospheric namelist needed editing for use in our environment. Atmospheric and surface forcings are included in the MUSC input files and the namelists in the util/musc/test/armcu directory are set up specifically for this case and are hence not edited by musc_run.sh.

Run your experiment

cd $HOME/hm_home/test_0001
./musc_run.sh -h
./musc_run.sh -d $HOME/muscCY46InputData/ARMCU_HAR -n ARMCU (There are now 4 ARMCU experiments to chose from e.g. ARMCU_EB and ARMCUs_EB are ones that use AROME namelists, ARMCU_Har and ARMCUs_Har use HARMONIE-AROME namelists. Currently, the results are a bit different.)

MUSC Output

DDH Toolbox

The outputs from a MUSC run are small files in lfa format. DDH tools can be used to handle these files.

To download the DDH toolbox, go to https://www.umr-cnrm.fr/gmapdoc/spip.php?article19 and download the tarball. Untar it and within the tools folder run ./install. Now the various "tools" are compiled. For example lfaminm $file shows you the max, min and mean of all the output variables in a file. lfac $file $var shows the value(s) of $var in $file e.g. lfac Out.000.0000.lfa PTS shows you surface temperature. In order to be able to use the plotting scripts below, you'll need the lfac tool in your path.

For example on ATOS, I set the following paths (may differ a bit for you depending on where you downloaded the ddhtools to). Perhaps add to your .bashrc file:

export PATH=$HOME/ddhtoolbox/tools/lfa/:$PATH
export DDHI_BPS=$HOME/ddhtoolbox/ddh_budget_lists/
export DDHI_LIST=$HOME/ddhtoolbox/ddh_budget_lists/conversion_list

Plot output time-series from the MUSC output lfa files

cd $HOME/hm_home/test_0001
./musc_plot1Dts.sh -d <musc-data-dir>

## python based plotting scripts and "default" png plots 
## will be produced in $HOME/hm_home/test_0001/plots1Dts

Extract output from the MUSC output `ICM*` fa files and plot time-series using these

By default you get ICM* files on the hour - you can change the namelist should you require a higher frequency.

cd $HOME/hm_home/test_0001
./musc_convertICM2ascii.sh -l <number_hours_in_expt_run> -f <path_to_your_musc_output>

## Generates an OUT ascii file for each atm and sfx ICM* input file
## ICM files have additional input not in lfa files e.g. TKE which is useful - also similar to 3D outputs

./musc_plot_profiles_ICMfiles.sh -d <data-dir> -p <parameter-model-level> -l <run_length_hours>

Creating your own input files

A converter script, musc_convert.sh, is available to extract a MUSC column from a model state file (ICMSHHARM+HHHH). musc_convert.sh is a Bash script that calls gl_grib_api to carry the data conversions.

Extract a MUSC input file

cd $HOME/hm_home/test_0001
./musc_convert.sh -d $HOME/muscCY46InputData/harm_arome/ -c extr3d -n REFIRL -l 53.5,-7.5 -t 6
mkdir $HOME/muscCY46InputData/musc_refirl
cp MUSCIN_REFIRL_atm.fa MUSCIN_REFIRL_sfx.fa MUSCIN_REFIRL_pgd.fa $HOME/muscCY46InputData/musc_refirl/

Convert MUSC FA to MUSC ASCII

cd $HOME/hm_home/test_0001
./musc_convert.sh -c fa2ascii -d $HOME/muscCY46InputData/musc_refirl -n REFIRL
ls -ltr
cp MUSCIN_REFIRL_atm.ascii MUSCIN_REFIRL_sfx.ascii $HOME/muscCY46InputData/musc_refirl/

Convert MUSC ASCII to MUSC FA

cd $HOME/hm_home/test_0001
./musc_convert.sh -c ascii2fa -d $HOME/muscCY46InputData/musc_refirl -n REFIRL
ls -ltr

Forcing in MUSC

musc_convert.sh includes forcing for temperature (11), humidity (51) and wind speed (32) .

You may edit the following lines to include other forcing:

  PPPKEY(1:4)%shortname  = 'ws','#','#','#',
  PPPKEY(1:4)%faname  = '#','SNNNFORC001','SNNNFORC002','SNNNFORC003'
  PPPKEY(1:4)%levtype = 'hybrid','hybrid','hybrid','hybrid',
  PPPKEY(1:4)%level   = -1,-1,-1,-1,
  PPPKEY(1:4)%pid     = 32,-1,-1,-1,
  PPPKEY(1:4)%nnn     = 0,0,0,0,
  PPPKEY(1:4)%lwrite  = F,T,T,T,
  IFORCE              = 11,51,32,

Further information on forcing is available here: MUSC/Forcing

MUSC local adaptation

KNMI workstations

The following files were added to make it possible to run MUSC on KNMI workstations:

config-sh/config.LinuxPC-MPI-KNMI
config-sh/submit.LinuxPC-MPI-KNMI
util/makeup/config.linux.gfortran.mpi-knmi

for use with the setup script:

./musc_setup.sh [...] -c LinuxPC-MPI-KNMI

In addition, the following workaround has to be applied to be able to run the REFL65 test case:

$ git diff src/ifsaux/utilities/echien.F90
diff --git a/src/ifsaux/utilities/echien.F90 b/src/ifsaux/utilities/echien.F90
index 55d5ce94e..694c87d83 100644
--- a/src/ifsaux/utilities/echien.F90
+++ b/src/ifsaux/utilities/echien.F90
@@ -532,7 +532,7 @@ IF((KINF == 0).OR.(KINF == -1).OR.(KINF == -2).OR.(KINF == -3)) THEN
            & 'LEVEL ',JFLEV,' : ',&
            & 'FILE = ',ZVALH(JFLEV), ' ; ARGUMENT = ',PVALH(JFLEV)
           IERRA=1
-          IERR=1
+!         IERR=1
         ENDIF
         IF(ABS(ZVBH(JFLEV)-PVBH(JFLEV)) > PEPS) THEN
           WRITE(KULOUT,*) ' VERTICAL FUNCTION *B* MISMATCH ON ',&

Then you are ready to compile:

remove the file experimentislocked from the experiment directory.
remove the directory with your previous build (if any).
start the compile with the musc_compile.sh script

When starting the MUSC run, add the PATH to mpirun and the libraries:

export PATH=$PATH:/usr/lib64/openmpi/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib64/openmpi/lib
./musc_run.sh [...]

MUSC FAQ

If there is an error, what files do I look in? NODE.001_01 and lola in your output directory.
How to I handle the output files? The output files are of the form Out.XXX.XXXX and appear in your output directory. There are in lfa format and can be handled using ddh tools. See the bash script musc_plot1Dts.sh for ideas. There are also ICM*lfa output files that are also handy for plotting profiles - use musc_convertICM2ascii.sh to convert these files to ASCII and musc_plot_profiles_ICMfiles.sh to plot some profiles e.g. TKE, cloud liquid etc.
I ran a different idealised case but did not get different results? The likely reason for this is that you did not delete the namelists from your experiment directory. If the namelists are there, the musc_run.sh script neither creates them nor copies them from the repository.
How do I create a new idealised case? This is not straightforward but the following was used to create the ASTEX cases in cy43 using info from cy38: https://www.overleaf.com/7513443985ckqvfdcphnng
How can I access a list of MUSC output parameters? Ensure you have the ddhtoolbox compiled. Then use lfaminm $file on any of your output files and it will show what is there. To look at a particular variable try lfac $file $parameter e.g. lfac $file PTS (for surface temperature). You can use cat to copy the values to an ASCII file for ease of use (e.g. lfac $file PTS > $ASCIIfile).
Is MUSC similar to the full 3D model version - is the physics the same? Yes, if you checkout develop then you have MUSC up-to-date with that.
Do I need to recompile the model if I modify code? Yes, if you modify code in a single file you must recompile the code but do not delete the original compiled model first. This will recompile relatively quickly. If you modify code in multiple files and you change what variables are passed between files, then you must delete your original compiled model and recompile the code. This will take longer to recompile.

MUSC variable names

A list of variable names found in the MUSC lfa output files can be found here. Please note that this is not a complete list of MUSC output parameters (yet). The variables in regular ICMSH... fa output are documented here

Outstanding Issues

ARMCU and Jenny's cases run without surface physics, radiation etc and hence return NANs in apl_arome. To circumvent this on ECMWF, we needed to compile less strictly. This needs to be investigated further.
The ASTEX cases currently do not run on ECMWF but work perfectly at Met Eireann - debugging needed.

MUSC using EMS

These instructions have moved to MUSC EMS