Choosing a Broker¶
The first step in getting Girder Worker up and running is installing a broker. The broker is a message queue such as RabbitMQ which receives messages and passes them to workers to execute a task. If you are running on an Ubuntu or Debian server you can install RabbitMQ with the following command
$ sudo apt-get install rabbitmq-server
Alternately, if you have docker installed, you can run the rabbitmq inside a container
$ docker run --net=host -d rabbitmq:latest
Installing Girder Worker¶
Girder Worker is a python package and may be installed with pip
$ pip install girder-worker
We recommend installing in a virtual environment to prevent package collision with your system Python.
Creating a Task Plugin¶
Task plugins are python packages. Multiple tasks may be placed in the same package but they must be installed in your environment to be discovered. Python packages require a certain amount of boilerplate to get started. The easiest way to create a package with a task plugin is to use the cookiecutter tool along with the Girder Worker plugin cookiecutter template.
First install cookiecutter
$ pip install cookiecutter
Next generate a task plugin Python package
$ cookiecutter gh:girder/cookiecutter-gw-plugin
This will prompt you with a number of questions about the package. For now you can simply select the defaults by hitting
Enter. This should create a
gw_task_plugin folder in your current working directory.
Adding Task Code¶
gw_task_plugin/gw_task_plugin/tasks.py file. You will find the following code.
from girder_worker.app import app from girder_worker.utils import girder_job # TODO: Fill in the function with the correct argument signature # and code that performs the task. @girder_job(title='Example Task') @app.task(bind=True) def example_task(self): pass
Edit example_task function to return the value “Hello World!”.
Installing the Task Plugin¶
The cookiecutter template has created a barebones Python package which can now be installed with pip. Return to the folder with the outermost
gw_task_plugin folder and install the package
$ pip install gw_task_plugin/
Running the Worker¶
Now run the worker from a command line
$ celery worker -A girder_worker.app -l info
If all is well, you should see a message similar to the following
-------------- celery@isengard v4.1.0 (latentcall) ---- **** ----- --- * *** * -- Linux-4.15.5-1-ARCH-x86_64-with-glibc2.2.5 2018-02-27 19:28:07 -- * - **** --- - ** ---------- [config] - ** ---------- .> app: girder_worker:0x7f72fd800ed0 - ** ---------- .> transport: amqp://guest:**@localhost:5672// - ** ---------- .> results: amqp:// - *** --- * --- .> concurrency: 4 (prefork) -- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker) --- ***** ----- -------------- [queues] .> celery exchange=celery(direct) key=celery [tasks] . girder_worker.docker.tasks.docker_run . gw_task_plugin.tasks.example_task [2018-02-27 19:28:07,205: INFO/MainProcess] Connected to amqp://guest:**@127.0.0.1:5672// [2018-02-27 19:28:07,226: INFO/MainProcess] mingle: searching for neighbors [2018-02-27 19:28:08,266: INFO/MainProcess] mingle: all alone [2018-02-27 19:28:08,321: INFO/MainProcess] celery@isengard ready.
As long as
gw_task_plugin.tasks.example_task is listed under the
[tasks] section then you are ready to move on to the next section.
Executing the Task¶
In a separate terminal, open up a python shell and type the following:
Import the task:
>>> from gw_task_plugin.tasks import example_task
Execute the task asynchronously:
>>> a = example_task.delay() >>> a.get() u'Hello World!'
In this tutorial we briefly demonstrated how to:
- Install and run a broker
- Install Girder Worker
- Create and install a task plugin
- Execute the task remotely with a Python interpreter
The goal here was to get up and running as quickly as possible and so each of these topics has been treated lightly.
- Celery supports a few different brokers. For more information see Celery’s complete broker documentation.
- Task plugin Python packages do more than just add a
setup.pyand create a
tasks.pyfor dumping tasks into. For more information on what the boilerplate the cookiecutter created see Plugins.
- Girder Worker aims to provide task execution API that is exactly the same as Celery. For more information on calling tasks see Celery’s Calling Tasks documentation. For more information about the knobs and dials available for changing how task execute, see Celery’s Task documentation.
Finally, we highly recommend reading through the Celery’s First Steps with Celery documentation as well as their User Guide. For some important differences between Celery and Girder Worker, we recommend keeping the Important Differences between Celery and Girder Worker page open while working through Celery’s documentation.