Running PyRate on a HPC system

This is a quick guide to getting the PyRate software up and running in a Portable Batch System (PBS) batch environment with MPI support. This setup is common in High Performance Compute (HPC) systems such as the National Computational Infrastructure's Raijin system.

The instructions below are tailored to the NCI Raijin system. Some instructions may not be applicable depending on the setup of the HPC system you are using. They should apply to both single-node and multi-node jobs; just set the number of cpus in the PBS directives in the job submission script accordingly (e.g. ncpus=32 for 2 nodes).

These instructions assume you are using bash shell.

Pre-installation

Note: These instructions currently only work with gcc and not Intel compilers.

1. Clone the PyRate repository into your home directory, or another directory of your choice:

$ cd ~
$ git clone git@github.com:GeoscienceAustralia/PyRate.git

Unload the icc compiler and default openmpi from the terminal:

$ module unload intel-cc
$ module unload intel-fc
$ module unload openmpi

Load the modules required for installation and running:
```
$ module load python3/3.4.3 python3/3.4.3-matplotlib
$ module load hdf5/1.8.10 gdal/2.0.0 openmpi/1.8 netcdf/4.3.2
```
(Alternatively, you may wish to add the above lines to your ~/.profile file)

Now add the following lines to the end of your ~/.profile file:

export PATH=$HOME/.local/bin:$PATH
export PYTHONPATH=$HOME/.local/lib/python3.4/site-packages:$PYTHONPATH
export PYRATEPATH=~/PyRate
export VIRTUALENVWRAPPER_PYTHON=/apps/python3/3.4.3/bin/python3
export LC_ALL=en_AU.UTF-8
export LANG=en_AU.UTF-8
source $HOME/.local/bin/virtualenvwrapper.sh

Install virtualenv and virtualenvwrapper in a terminal:

$ pip3 install  --user virtualenv virtualenvwrapper

Refresh your environment by sourcing your ~/.profile file:
```
$ source ~/.profile
```

Installation

Create a new virtualenv for PyRate:

$ mkvirtualenv --system-site-packages pyrate

Make sure the virtualenv is activated:
```
$ workon pyrate
```

Install pyrate:

$ cd $PYRATEPATH
$ pip install python-daemon==2.1.1
$ python setup.py install

Once installation has completed, you can run the tests to verify everything has gone correctly:
```
$ pip install pytest
$ pytest ~/PyRate/tests/
```

Updating the Code

To update the PyRate code, first make sure you are in the pyrate virtual environment:

$ workon pyrate

Next, pull the latest commit from the master branch, and install:

$ cd $PYRATEPATH
$ git pull origin
$ python setup.py install

If the pull and the installation complete successfully, the code is ready to run!

Running Batch Jobs

An example script to assist launching batch jobs over multiple nodes with PBS is given at the bottom of this readme.

Batch testing

To check everything is working, submit the tests as a batch job:

$ cd $PYRATEPATH/pbs
$ qsub submit_tests.sh

MPIRun

PyRate uses MPI internally for parallelization. To run a script or demo, use the command:

$ mpirun -n <num_procs> <command>

For example:

$ mpirun -n 16 pyrate prepifg pyrate_pbs.conf

A PBS job submission script might look like this:

#!/bin/bash
#PBS -P <project>
#PBS -q <queue>
#PBS -l walltime=01:00:00,mem=128GB,ncpus=16,jobfs=20GB
#PBS -l wd
#PBS -j oe

# setup environment
module unload intel-cc
module unload intel-fc
module load python3/3.4.3 python3/3.4.3-matplotlib
module load load hdf5/1.8.10 gdal/2.0.0
source $HOME/.profile

# start the virtualenv
workon pyrate

# run PyRate commands
mpirun --mca mpi_warn_o pyrate prepifg /path/to/config_file.conf
mpirun --mca mpi_warn_o pyrate linrate /path/to/config_file.conf
mpirun --mca mpi_warn_o pyrate postprocess /path/to/config_file.conf

--mca mpi_warn_o prevents mpi fork warnings. To distribute the job between nodes and to manage memory between processes use something like -map-by ppr:4:node, which will start 4 jobs per node.

update version to 0.2.0

README.rst