scheduler_tools

Tools for composing interdependent tasks on a scheduler (i.e. SLURM)

Setup requirements for launching prefect / dask-distributed on the cluster from your local desktop

On your local system

For whatever project you are working on add scheduler_tools to your dependency list.

conda activate {your_env_name}
pip install -e {your_project_name}
mkdir ~/.aics_dask

create file ~/.aics_dask/ssh.json with contents

{"gateway":{
	"url": "slurm-machine.bettertech.com",
	"user": "flanders",
	"identityfile": "/Users/flanders/ssh/flandersPrivateKey"
	},
    "dashboard_port": 8787,
    "dask_port": 34009
}

On the cluster

conda activate {your_env_name}
pip install -e {your_project_name}
mkdir ~/.aics_dask

programmatically launch remote server and create tunnel

from scheduler_tools import Connector
from pathlib import Path

dask_prefs = dict()
dask_prefs['queue'] = 'aics_cpu_general'
dask_prefs['cores'] = 2
dask_prefs['ram'] = '8GB'
dask_prefs['wall_time'] = "02:00:00"
dask_prefs['min_workers'] = 2
dask_prefs['max_workers'] = 40
dask_prefs['remote_env'] = 'dask-scheduler'
dask_prefs['remote_command'] = 'setup_and_spawn.bash'
dask_prefs['remote_path'] = '~/.aics_dask/'

conn = Connector(distributed_dict=dask_prefs, pref_path=Path('~/.aics_dask/ssh.json'))
conn.run_command()
conn.stop_forward_if_running()
conn.forward_ports()

programmatically shutdown remote server and tunnel

from scheduler_tools import Connector
from pathlib import Path

conn = Connector(pref_path=Path('~/.aics_dask/ssh.json'))
conn.stop_dask()

Command line interface

CLI start Dask Cluster command

spawn_dask_cluster -s ~/.aics_dask_gpu/ssh.json -q aics_gpu_general -r <remote_env_name>

CLI stop Dask Cluster command

shutdown_dask_cluster -s ~/.aics_dask/ssh.json

Free software: Allen Institute Software License