API documentation

Core

class girder_worker.GirderWorkerPlugin(app, *args, **kwargs)[source]
task_imports()[source]

Plugins must override this method.

class girder_worker.GirderWorkerPluginABC(app, *args, **kwargs)[source]

Abstract base class for Girder Worker plugins. Plugins must descend from this class; see the Plugins section for more information.

task_imports()[source]

Plugins must override this method.

class girder_worker.task.Task[source]

Girder Worker Task object. Tasks defined by plugins must be subclasses of this class, however you will typically not need to reference it yourself, as it will be automatically instantiated by the girder_worker celery app. See Creating tasks for instructions.

canceled

A property to indicate if a task has been canceled.

Returns:True is this task has been canceled, False otherwise.
Return type:bool

Transforms

class girder_worker_utils.transforms.girder_io.GirderUploadJobArtifact(job_id=None, name=None, **kwargs)[source]

This class can be used to upload a directory of files or a single file as artifacts attached to a Girder job. These files are only uploaded if they exist, so this is an optional output.

Currently, only a flat directory of files is supported; the transform does not recurse through nested directories, though that may change in the future.

Docker

Tasks

Transforms

class girder_worker.docker.transforms.BindMountVolume(host_path, container_path, mode='rw')[source]

A volume that will be bind mounted into a docker container.

Parameters:
  • host_path (str) – The path on the host machine.
  • container_path (str) – The path in the container this volume will be mounted at.
  • mode (str) – The mounting mode
class girder_worker.docker.transforms.ChunkedTransferEncodingStream(url, headers={}, **kwargs)[source]

A stream transform that allows data to be streamed using HTTP Chunked Transfer Encoding to a server.

Parameters:
  • url (str) – Destination URL for the stream.
  • headers – HTTP headers to send.
class girder_worker.docker.transforms.Connect(input, output)[source]

This utility class represents the connection between a girder_worker.docker.transforms.NamedOutputPipe or girder_worker.docker.transforms.NamedInputPipe and one of the other streaming transforms. Girder Worker will stream the data to or from the named pipe.

Parameters:
class girder_worker.docker.transforms.ContainerStdErr[source]

Represents the standard error stream of the container. Can be used with girder_worker.docker.transforms.Connect to redirect the containers standard error to another stream.

class girder_worker.docker.transforms.ContainerStdOut[source]

Represents the standard output stream of the container. Can be used with girder_worker.docker.transforms.Connect to redirect the containers standard output to another stream.

class girder_worker.docker.transforms.HostStdErr[source]

Represents the standard error stream on the host machine. Can be used with girder_worker.docker.transforms.Connect to write text to stderr.

class girder_worker.docker.transforms.HostStdOut[source]

Represents the standard output stream on the host machine. Can be used with girder_worker.docker.transforms.Connect to write text to stdout.

class girder_worker.docker.transforms.NamedInputPipe(name, container_path=None, host_path=None, volume=<girder_worker.docker.transforms._DefaultTemporaryVolume object>)[source]

A named pipe that can be open for read within a docker container. i.e. To stream data into a container.

Parameters:
  • name (str) – The name of the pipe.
  • container_path (str) – The path in the container.
  • host_path (str) – The path on the host machine.
  • volume – Alternatively a girder_worker.docker.transforms.BindMountVolume instance can be provided. In which case the container_path and host_paths from the volume will be used when creating the pipe. The default location is girder_worker.docker.transforms.TemporaryVolume.default
class girder_worker.docker.transforms.NamedOutputPipe(name, container_path=None, host_path=None, volume=<girder_worker.docker.transforms._DefaultTemporaryVolume object>)[source]

A named pipe that can be opened for write within a docker container. i.e. To stream data out of a container.

Parameters:
  • name (str) – The name of the pipe.
  • container_path (str) – The path in the container.
  • host_path (str) – The path on the host machine.
  • volume – Alternatively a girder_worker.docker.transforms.BindMountVolume instance can be provided. In which can the container_path and host_paths from the volume will be use when creating the pipe. The default location is girder_worker.docker.transforms.TemporaryVolume.default
class girder_worker.docker.transforms.TemporaryVolume(host_dir=None, mode=493)[source]

This is a class used to represent a temporary directory on the host that will be mounted into a docker container. girder_worker will automatically attach a default temporary volume. This can be reference using TemporaryVolume.default class attribute. A temporary volume can also be create in a particular host directory by providing the host_dir param.

Parameters:
  • host_dir (str) – The root directory on the host to use when creating the the temporary host path.
  • mode (int) – The default mode applied to the temporary volume if it does not already exist.
class girder_worker.docker.transforms.VolumePath(filename, volume=<girder_worker.docker.transforms._DefaultTemporaryVolume object>)[source]

A path on a docker volume. Must be a path relative to the root of the volume.

Parameters:
class girder_worker.docker.transforms.girder.GirderFileIdToStream(_id, **kwargs)[source]

This can be used to stream a Girder file into a docker container. See Streaming Girder files into a container for example usage.

Parameters:_id (str or ObjectId) – The Girder file ID.
class girder_worker.docker.transforms.girder.GirderFileIdToVolume(_id, volume=<girder_worker.docker.transforms._DefaultTemporaryVolume object>, filename=None, **kwargs)[source]

This can be used to pass a Girder file into a docker container. It downloads the file to a bind mounted volume, and returns the container path of the file.

Parameters:
  • _id (str or ObjectId) – The Girder file ID.
  • volume (girder_worker.docker.transforms.BindMountVolume) – The bind mount volume where the file will reside.
  • filename (str) – Alternate name for the file. Default is to use the name from Girder.
class girder_worker.docker.transforms.girder.GirderFolderIdToVolume(_id, volume=<girder_worker.docker.transforms._DefaultTemporaryVolume object>, folder_name=None, **kwargs)[source]

This can be used to pass a Girder folder into a docker container. It downloads the folder to a bind mounted volume, and returns the container path of the directory.

Parameters:
  • _id (str or ObjectId) – The Girder folder ID.
  • volume (girder_worker.docker.transforms.BindMountVolume) – The bind mount volume where the directory will reside.
  • folder_name (str) – Alternate name for the directory. Default is to use the name from Girder.
class girder_worker.docker.transforms.girder.GirderItemIdToVolume(_id, volume=<girder_worker.docker.transforms._DefaultTemporaryVolume object>, **kwargs)[source]

This can be used to pass a Girder item into a docker container. It downloads the item to a bind mounted volume, and returns the container path of the directory.

Parameters:
  • _id (str or ObjectId) – The Girder item ID.
  • volume (girder_worker.docker.transforms.BindMountVolume) – The bind mount volume where the item will reside.
  • item_name (str) – Alternate name for the file. Default is to use the name from Girder.
class girder_worker.docker.transforms.girder.GirderUploadVolumePathJobArtifact(volumepath, job_id=None, name=None, upload_on_exception=False, **kwargs)[source]

This transform can be used to upload artifacts produced during a docker task execution and attach them to the corresponding job in Girder. This can be useful for tracing and debugging jobs, or simply collecting intermediate information during job execution. If the passed in path does not exist, this is a no-op.

Parameters:
  • volumepath (girder_worker.docker.transforms.VolumePath) – A volume path pointing to a mounted directory or file. If a directory, all files within the directory will be uploaded as artifacts to the job. If a file, just uploads the single file. If it does not exist, no action is performed.
  • job_id (str) – The job ID to attach the artifacts to. If calling this from Girder via docker_run.delay, you will not need to set this, as it will be set automatically.
  • name (str) – A name for the artifact. Only applies for single file paths. If not specified, will use the basename of the file.
  • upload_on_exception (bool) – If True, this transform will occur even if the docker task fails. This can be used to debug failed docker_run tasks.
class girder_worker.docker.transforms.girder.GirderUploadVolumePathToFolder(volumepath, folder_id, delete_file=False, **kwargs)[source]

This transform uploads data in a bind mount volume to a Girder folder. This should be used in girder_result_hooks to upload data produced by the task.

Parameters:
  • volumepath (girder_worker.docker.transforms.VolumePath) – The location of the file or directory to upload.
  • folder_id (str or ObjectId) – The folder ID in Girder.
  • delete_file (bool) – Whether to delete the data afterward.
class girder_worker.docker.transforms.girder.GirderUploadVolumePathToItem(volumepath, item_id, delete_file=False, **kwargs)[source]

This transform uploads data in a bind mount volume to a Girder item. This should be used in girder_result_hooks to upload files produced by the task.

Parameters:
  • volumepath (girder_worker.docker.transforms.VolumePath) – The location of the file to upload.
  • item_id (str or ObjectId) – The item ID in Girder.
  • delete_file (bool) – Whether to delete the file afterward.
class girder_worker.docker.transforms.girder.ProgressPipe(name='.girder_progress', volume=<girder_worker.docker.transforms._DefaultTemporaryVolume object>)[source]

This can be used to stream progress information out of a running docker container as part of a docker_run task. For a usage example, see Streaming progress reporting from Docker tasks to Girder jobs.

Parameters: