The HARMONIE testbed

The HARMONIE testbed provides a facility to run a number of well defined test cases using the existing script environment in HARMONIE. The ALADIN testbed, Mitraillette runs test on the hart of the model, the dynamical core. The HARMONIE testbed tests the full script system as it is supposed to be used.

Defining the configurations

General

The testbed is a suite that launches and follows new experiments one at a time in a controlled environment. The testbed experiment takes care of compilation and also hosts the climate files generated by the tested configurations. Source and scripts changes shall be done in the testbed experiment and will be synchronized to the child experiment using the hm_CMODS option in HARMONIE.

A number of basic configurations have been defined in scr/Harmonie_configurations.pm as the deviation from the default setup in ecf/config_exp.h, scr/include.ass and suites/harmonie.pm. These configurations are controlled by the script scr/Harmonie_testbed.pl. The script also contains a number of extra configurations tested from time to time. With the current settings, a test of AROME without 3DVAR would look like.

    # AROME no 3D-VAR but default blending of upper air from boundaries
    'AROME' => {
      'description' => 'Standard AROME settings without upper air DA',
      'PHYSICS'     => 'arome',
      'SURFACE'     => 'surfex',
      'DYNAMICS'    => 'nh',
      'ANAATMO'     => 'blending',
      'ANASURF'     => 'none',
      'DFI'         => 'none',
      'HOST_MODEL'  => 'ifs',
      'DOMAIN'      => 'DKCOEXP',
      'VLEV'        => '65',
      'BDINT'       => '3',
    },

The resulting output, in this case from AROMEBDARO running at ECMWF would look like:

 Using the configuration AROME_BD_ARO


 Input /scratch/hlam/hm_home/testbed_ECMWF.atos.gnu_12413/ecf/config_exp.h  
 Output ecf/config_exp.h  

 Change BUILD=${BUILD-yes}                     to BUILD=no 
 Change       BUILD_ROOTPACK=no to BUILD_ROOTPACK=no 
 Change BINDIR=${BINDIR-$HM_DATA/bin}                 to BINDIR=$HM_COMDAT/testbed_ECMWF.atos.gnu_12413/bin 
 Change DOMAIN=DKCOEXP                          to DOMAIN=TEST_8 
 Change VLEV=65                                 to VLEV=HIRLAM_60 
 Change LL=${LL-06}                             to LL=6 
 Change DYNAMICS="nh"                           to DYNAMICS=nh 
 Change PHYSICS="arome"                         to PHYSICS=arome 
 Change SURFACE="surfex"                        to SURFACE=surfex 
 Change DFI="none"                              to DFI=none 
 Change ANAATMO=3DVAR                           to ANAATMO=none 
 Change ANASURF=CANARI_OI_MAIN                  to ANASURF=none 
 Change OBDIR=$HM_DATA/observations             to OBDIR=$HM_DATA/../testbed_ECMWF.atos.gnu_12413/observations/ 
 Change HOST_MODEL="ifs"                        to HOST_MODEL=aro 
 Change HOST_SURFEX="no"                        to HOST_SURFEX=yes 
 Change SURFEX_INPUT_FORMAT=lfi                 to SURFEX_INPUT_FORMAT=fa 
 Change BDLIB=ECMWF                             to BDLIB=AROME 
 Change BDDIR=$HM_DATA/${BDLIB}/archive/@YYYY@/@MM@/@DD@/@HH@   to BDDIR=$HM_DATA/../testbed_ECMWF.atos.gnu_12413/archive_AROME/@YYYY@/@MM@/@DD@/@HH@/ 
 Change INT_BDFILE=$WRK/ELSCF${CNMEXP}ALBC@NNN@                 to INT_BDFILE=$ARCHIVE_ROOT/$YY/$MM/$DD/$HH/ELSCF${CNMEXP}ALBC@NNN@ 
 Change BDSTRATEGY=simulate_operational to BDSTRATEGY=same_forecast 
 Change BDINT=1                         to BDINT=3 
 Change SURFEX_PREP="no"                to SURFEX_PREP=yes 
 Change CLIMDIR=$HM_DATA/climate                to CLIMDIR=$HM_DATA/../testbed_ECMWF.atos.gnu_12413/climate/$DOMAIN/$PHYSICS 
 Change BDCLIM=$HM_DATA/${BDLIB}/climate        to BDCLIM=$HM_DATA/../testbed_ECMWF.atos.gnu_12413/climate/TEST_11/arome/ 
 Change INT_SINI_FILE=$WRK/SURFXINI.$SURFEX_OUTPUT_FORMAT       to INT_SINI_FILE=$ARCHIVE_ROOT/$YY/$MM/$DD/$HH/SURFXINI.$SURFEX_OUTPUT_FORMAT 
 Change ARCHIVE_ECMWF=yes                       to ARCHIVE_ECMWF=no 
 Change POSTP="inline"                          to POSTP=inline 
 Change MAKEGRIB=no                             to MAKEGRIB=yes 
 Add new settings JBDIR=$HM_REV/testbed_data/jb_data 
 Add new settings LARGE_EC_BD=no 
 Add new settings PLAYFILE=harmonie 
 Add new settings config=AROME 


 Input /scratch/hlam/hm_home/testbed_ECMWF.atos.gnu_12413/Env_submit 
 Output Env_submit 

 Change $nprocx=16; to $nprocx=2 
 Change $nprocy=16; to $nprocy=2 
 Change $nprocx=8; to $nprocx=2 
 Change $nprocy=16; to $nprocy=2 

As seen from the example above the script also changes the submission rules. These rules can be defined, per host, at the end of the script. Other host specific settings may also be defined to allow local changes of the test environment. In Harmonie_testbed.pl we find e.g. changes for the default at ECMWF:

 'default' => {
   'BINDIR'     => '$HM_COMDAT/'.$EXP.'/bin',
   'BDDIR'      => '$HM_DATA/../'.$EXP.'/$BDLIB/$DOMAIN',
   'OBDIR'      => '$HM_DATA/../'.$EXP.'/observations/$DOMAIN',
   'LARGE_EC_BD' => 'no',
   'CLIMDIR'    => '$HM_DATA/../'.$EXP.'/climate/$DOMAIN/$PHYSICS',
  },

The host dependent settings will be imposed on all configurations. If a setting in any configuration is in conflict with the host settings the configuration settings will be used.

Define changes in harmonie.pm

The changes to msms/harmonie.pm are controlled with a special syntax, like in the AROME_JB configuration.

   # AROME Structure function derivation
   'AROME_JB' => {
     'description' => 'Derive structure functions for AROME 3DVAR',
      ...
      'harmonie.pm' => ['ENSBDMBR','ENSCTL','SLAFLAG'],
        'ENSBDMBR'    => '[1,2,3,4]',
        'ENSCTL'      => '["001","002","003","004"]',
        'SLAFLAG'     => '[0]',
    },

The harmonie.pm key determines which keyword to find and replace. The list guarantees that the same keywords are not changed in e.g. ecf/config_exp.h .

Testbed members

Testbed domains

The playfile

The playfile used for the testbed is msms/testbed.tdf. Here each configuration is defined with a trigger, a task to create and one to follow the child experiments. Configurations inluded are listed in the TESTBED_LIST environment variable in ecf/config_exp.h

 TESTBED_LIST="ALADIN ALADIN_3DVAR AROME"

If the child experiment fails the Follow_exp task will also fail. When the child experiment problem has been corrected and the task restarted, the follow task should be restarted. When, finally, the child experiment is finished the test family will be completed and next test case will be triggered.

Input data

The standard testbed configuration is run over several. The domain and resolution is chosen to be computationally cheap and not to give meteorologically interesting and meaningful results. Input data for running the testbed is ECMWF boundaries, observations, background error statistics and climate files. The latest data may be found on /scratch/hlam/hm_home/some_recent_testbed_experiment/testbed_data and the data included is the following:

• ECMWF boundaries for the tested periods
• Observations
• Background errors
• Climate files
• Forcing data for nested experiments

Download the data to your machine and put it on the default location $HM_REV/testbed_data or define your location in scr/Harmonie_testbed.pl. If you wish to test the climate generation you simple redefine the location of the climate files or remove the existing ones in the climate directory of the testbed. The testbed data typically includes the following:

Starting the testbed

The testbed experiment is setup as any normal experiment with

Harmonie setup -r REVISION -h HOST

The testbed is launched by

Harmonie testbed 

Before you start the testbed you should define your reference experiment. The reference experiment is picked automatically as an experiment with the same name but with lower revision number. The reference experiment can also be defined by the by setting REFEXP in ecf/config_exp.h as the full path to another testbed experiment:

export REFEXP=/scratch/hlam/hm_home/test_37h12

Evaluation of the result

At the end of each testbed run the results are compared to a reference experiment using the script scr/Testbed_copmp. The script uses xtool to check the numerical difference for the following output:

• ECMWF input data
• climate files
• Interpolated boundary files
• Output forecast files in FA format
• Output forecast files in GRIB format
• vfld/vobs files

In addition the internal consistency is checked by comparing runs with

• 1D vs 2D decomposition
• EPS control vs a deterministic run
• Run with and without IO-server

The choice of internal consistency tests reflects the history of problems and inconsistencies encountered.

HARMONIE testbed results from ecflow-gen-hlam-001
Sat Nov 16 20:41:27 GMT 2019

Configuration: ECMWF.atos.gnu

Compare experiment testbed_ECMWF.atos.gnu_12414 and testbed_ECMWF.atos.gnu_12413

Check:ecmwf_bd
  Output grib file summary (differ/missing/total) 0/0/37
   comparison took 164 seconds
 Configuration ecmwf_bd is equal

Check:climate
  Output internal file summary (differ/missing/total) 0/0/52
   comparison took 34 seconds
 Configuration climate is equal

Check:AROME_3DVAR
  Output internal file summary (differ/missing/total) 0/0/36
   comparison took 79 seconds
  Output grib file summary (differ/missing/total) 0/0/96
   comparison took 91 seconds
  vfld/vobs file summary (differ/missing/total) 0/0/36
   comparison took 4 seconds
 Configuration AROME_3DVAR is equal

 ...

Compare AROME_BD_ARO and AROME_BD_ARO_IO_SERV
 No differences found
   comparison took 423 seconds

Testbed comparison complete

[ Status: OK]

 For more details please check /scratch/hlam/hm_home/testbed_ECMWF.atos.gnu_12414/testbed_comp_12414.log_details

All the logs from a testbed experiment are posted to the mail address MAIL_TESTBED set in ecf_config_exp.h. If a github token GH_TOKEN is set in scr/Testbed_comp the results will also be posted on Testbed output discussions on github using the GraphQL API. See github settings to create a token. Tick at least the repo box. Save your token in $HOME/.ssh/gh_testbed.token or in $HOME/env/gh_testbed.token and chmod 600 and it will be used. The test returns three different status signals

• OK means that all configurations reproduces the result of your reference experiment.
• OK, BUT NO COMPARISON means that the suit run through but that there was nothing to compare with
• FAILED means that the internal comparisons failed
• DIFFER means that one more configurations differ from your reference experiment
• FAILED and DIFFER is a combination of the last two

In addition to the summary information detailed information can be found in the archive about the art of the difference.

When to use the testbed

It is recommended to use the testbed when adding new options or make other changes in the configurations. If your new option is not activated the result compared with the reference experiment should be the same, if not you have to start debugging. When changing things for one configuration it's easy to break other ones. In such cases the testbed is a very good tool make sure you haven't destroyed anything.