Plugins¶
Task Plugin from Cookiecutter¶
The simplest way to bootstrap your Girder Worker task plugin is to use our cookiecutter plugin to fill in the boilerplate. See Creating a Task Plugin for instructions.
Task Plugin from Scratch¶
This is an example plugin that demonstrates how to extend girder_worker
by allowing it to run additional tasks. Plugins are implemented as
separate pip installable packages. To install this example plugin you
can checkout this code base, change directories to
examples/plugin_example/
and run pip install .
This will add the
gwexample
plugin to girder_worker. If you then run girder_worker
with a log level of ‘info’ (e.g. girder-worker -l info
) you should
see the following output:
(girder)$ girder-worker -l info
-------------- celery@minastirith v3.1.23 (Cipater)
---- **** -----
--- * *** * -- Linux-4.8.6-1-ARCH-x86_64-with-glibc2.2.5
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app: girder_worker:0x7f69bfff1050
- ** ---------- .> transport: amqp://guest:**@localhost:5672//
- ** ---------- .> results: amqp://
- *** --- * --- .> concurrency: 32 (prefork)
-- ******* ----
--- ***** ----- [queues]
-------------- .> celery exchange=celery(direct) key=celery
[tasks]
. girder_worker.convert
. girder_worker.run
. girder_worker.validators
. gwexample.analyses.tasks.fibonacci
[2016-11-08 12:22:56,163: INFO/MainProcess] Connected to amqp://guest:**@127.0.0.1:5672//
[2016-11-08 12:22:56,184: INFO/MainProcess] mingle: searching for neighbors
[2016-11-08 12:22:57,198: INFO/MainProcess] mingle: all alone
[2016-11-08 12:22:57,218: WARNING/MainProcess] celery@minastirith ready.
Notice that the task gwexample.analyses.tasks.fibonacci
is now
available. With the girder-worker processes running, you should be able to
execute python example_client.py
in the current working directory.
After a brief delay, this should print out 121393
- the Fibonacci
number for 26.
Writing your own plugin¶
Adding additional tasks to the girder_worker infrastructure is easy and
takes three steps. (1) Creating tasks, (2) creating a plugin class and
(3) adding a girder_worker_plugins
entry point to your setup.py.
Creating tasks¶
Creating tasks follows the standard celery
conventions.
The only difference is the celery application that decorates the
function should be imported from girder_worker.app
. E.g.:
from girder_worker.app import app
@app.task
def fibonacci(n):
if n == 1 or n == 2:
return 1
return fibonacci(n-1) + fibonacci(n-2)
Plugin Class¶
Each plugin must define a plugin class the inherits from
girder_worker.GirderWorkerPluginABC
. GirderWorkerPluginABC’s
interface is simple. The class must define an __init__
function and
a task_imports
function. __init__
takes the girder_worker’s
celery application as its first argument. This allows the plugin to
store a reference to the application, or change configurations of the
application as necessary. The task_imports
function takes no
arguments and must return a list of the package paths (e.g. importable
strings) that contain the plugin’s tasks. As an example:
from girder_worker import GirderWorkerPluginABC
class GWExamplePlugin(GirderWorkerPluginABC):
def __init__(self, app, *args, **kwargs):
self.app = app
# Update the celery application's configuration
# it is not necessary to change the application configuration
# this is simply included to illustrate that it is possible.
self.app.config.update({
'TASK_TIME_LIMIT': 300
})
def task_imports(self):
return ['gwexample.analyses.tasks']
Entry Point¶
Finally, in order to make the plugin class discoverable, each plugin
must define a custom entry point in its setup.py
. For our example,
this entry point looks like this:
from setuptools import setup
setup(name='gwexample',
# ....
entry_points={
'girder_worker_plugins': [
'gwexample = gwexample:GWExamplePlugin',
]
},
# ....
)
Python Entry Points are a way for python packages to advertise classes and objects to other installed packages. Entry points are defined in the following way:
entry_points={
'entry_point_group_id': [
'entry_point_name = importable.package.name:class_or_object',
]
}
The girder_worker package introduces a new entry point group
girder_worker_plugins
. This is followed by a list of strings which
are parsed by setuptools. The strings must be in the form
name = module:plugin_class
Where name
is an arbitrary string (by
convention the name of the plugin), module
is the importable path to
the module containing the plugin class, and plugin_class
is a class
that inherits from GirderWorkerPluginABC
.
Final notes¶
With these three components (Tasks, Plugin Class, Entry Point) you should be able to add arbitrary tasks to the girder_worker client.
Writing cancelable tasks¶
girder_worker provides support for signaling that a task should be canceled using
Celery’s revoke
mechanism. In order for a task to be able to be canceled cleanly it must periodically
check if it has been canceled, if it has then is can do any necessary cleanup and
return. girder_worker provides a task base class (girder_worker.utils.Task
)
that provides a property that can be used to check if the task has been canceled.
An example of its use is shown below:
from girder_worker.app import app
@app.task(bind=True)
def my_cancellable_task(task):
while not task.cancelled:
# Do work
The Girder job model associated with the canceled task will be moved into the
JobStatus.CANCELED
state.