How to run on the AWS DP HPC cluster interactively
==================================================

The Rapthor Prefect prototype can be run on a single node on AWS using the 
example `aws-interactive.sh <http://gitlab.com/ska-telescope/sdp/ska-sdp-rapthor-prefect-prototype/-/blob/main/aws-interactive.sh>`_
shell script provided. This script will load dependencies available via spack 
and install additional python dependencies (via poetry) in a virtual environment. 
It will then start a local prefect server and run the code in 
`poc.py <http://gitlab.com/ska-telescope/sdp/ska-sdp-rapthor-prefect-prototype/-/blob/main/src/ska_sdp_rapthor_prefect_prototype/poc.py>`_. 
The script has been tested on the AWS DP HPC cluster using an interactive compute node and 8 CPUs.

- If you want to run inside a container on your local machine instead, 
  see the `quickstart guide <../quickstart.html>`_.
- If you want to run via a SLURM job on a single node instead using a prefect 
  server running on the headnode, see the 
  `instructions for running via SLURM <./aws_slurm.html>`_.

Prerequisites
-------------
- An account on the AWS DP HPC cluster
- Optional: SSH tunnel to the compute node you will be using to forward port 
  4200 (for the Prefect UI) and port 8787 (for the Dask dashboard) to your local 
  machine.

Steps
-----

#. Log into the DP HPC headnode. Information about the DP HPC cluster, including 
   how to gain access is available from `this confluence page 
   <https://confluence.skatelescope.org/display/SE/DP+AWS+HPC+Cluster+Information>`_.

#. Change to a directory where you want to clone the project.

#. Clone the project repository using git.

   .. code-block:: console

      git clone https://gitlab.com/ska-telescope/sdp/ska-sdp-rapthor-prefect-prototype.git

#. Change to the project directory.

   .. code-block:: console

      cd ska-sdp-rapthor-prefect-prototype

#. Optionally check out the branch of the repo that you want to test:
   
   .. code-block:: bash

      git checkout <feature-branch>

#. Start an interactive compute node.

   .. code-block:: console

      srun --nodes=1 --partition=any-r7i-24xl-spt --cpus-per-task=8 --ntasks-per-node=1 --time=1:00:00 --pty bash -i

#. If you want to use an SSH tunnel to access the Prefect UI and Dask 
   dashboard on your local machine, follow these additional steps:
    
    #. Get the hostname of the compute node and use this to set up port 
       forwarding from your local machine to the compute node. Copy the
       compute node hostname after running the command below.

       .. code-block:: console

          echo $HOSTNAME
    
    #. On your local machine, set up an SSH tunnel to the compute node to forward 
       port 4200 for the Prefect UI and port 8787 for the Dask dashboard.

       .. code-block:: console

          ssh -L 127.0.0.1:4200:<compute-node-hostname>:4200 -L 127.0.0.1:8787:<compute-node-hostname>:8787 dp-hpc-headnode

#. Edit the ``aws-interactive.sh`` script to set the correct path to the project directory.
   It will use the current directory by default.

#. Run the ``aws-interactive.sh`` script. This will install the project and dependencies via
   poetry, check the installation, and then run the `start.sh <../../../start.sh>`_
   script to launch the pipeline. The output of the script will be saved in a
   log file.

   .. code-block:: bash

      ./aws-interactive.sh | tee rpp-aws-`date +"%FT%T"`.log

#. The Prefect server will start automatically. The code in ``poc.py`` will run
   automatically when the script starts. If you have set up the SSH tunnel,
   once the Prefect server is up, you can 
   monitor the progress of the flow in the Prefect UI at 
   `<http://localhost:4200>`_  and the Dask dashboard at 
   `<http://localhost:8787>`_.

#. When the flow has completed, you can stop the prefect server by pressing 
   ``CTRL-C`` in the terminal where you ran the script.