🐍⚡️Support Python for everyone by grabbing a 30% discount on PyCharm—ALL proceeds go to the Python Software Foundation. Offer ends soon, so grab it today!  GET 30% OFF PYCHARM

Asynchronous and distributed scheduling of Ewoks workflows from python

Navigation

Verified details

These details have been verified by PyPI
Maintainers
Avatar for huderl from gravatar.com huderl Avatar for payno from gravatar.com payno Avatar for poautran from gravatar.com poautran Avatar for woutdenolf from gravatar.com woutdenolf

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: MIT License (MIT)
  • Author: ESRF
  • Requires: Python >=3.6
Classifiers
Report project as malware

Project description

ewoksjob

Utilities for job scheduling of ewoks workflows.

Ewoksjob provides an ewoks interface for asynchronous and distributed scheduling of ewoks from python.

Note that ewoksjob distributes the execution of workflows while ewoksdask distributes the execution of tasks in a workflow. So in the context of workflows, job scheduling exists on two levels.

The primary clients that need to schedule workflows are

Installation

Install on the client side

pip install ewoksjob

Install on the worker side

pip install ewoksjob[worker]

Getting started

Start a worker pool that can execute ewoks graphs

ewoksjob worker

Start a workflow on the client side

from ewoksjob.client import submit

workflow = {"graph": {"id": "mygraph"}}
future = submit(args=(workflow,))
result = future.get()

Note that both environments need to be able to import celeryconfig which contains celery configuration (mainly the message broker and result backend URL's).

Hello world example

Clone the git repository and start a worker pool

scripts/worker.sh --sql

Submit workflows

scripts/runjobs.sh --sql

Tests

pytest --pyargs ewoksjob

To run the redis tests you need redis-server (e.g. conda install redis-server).

Documentation

https://ewoksjob.readthedocs.io/

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page