A Task is the ClearML entity which can represent an experiment, a step in a workflow, a workflow controller, or any custom implementation you choose. For example, a Task can be an experiment, a Task can be an editable copy of another Task, a template for experiments, or a Task can control the orchestration of other Tasks: one Task can control hyperparameter optimization while another Task is the template for an experiment to optimize; or, one Task can control a pipeline composed of any number of other Tasks which accomplish work (e.g., data preprocessing, training, testing, and best model selection).
Creating new Tasks¶
Create Tasks several ways. For example, initialize a Task in code, clone a Task from an existing Task using the Web UI or programmatically, or create Tasks in automated workflows, running locally or on a remote machine.
Data stored in Tasks¶
Stored in a Task are the data needed for tracking and visualizing results; reproducing, tuning, and comparing experiments; and executing workflows. A Task contains the Python environment data (packages and versions), Git codebase data (repository, branch, commit ID, uncommitted changes), hyperparameters (whether they originate as command-line arguments, TensorFlow Definitions, or in parameter dictionaries), artifact storage locations (initial input weights model, output model, and model snapshots), debug sample storage locations, logs, metrics, and other results data.
Since a Task stores artifact and debug sample storage locations, those storage locations must be accessible when the Task executes. Otherwise, the Task cannot find the data. The same is true if your script loads data. That data must be accessible to the Task when it executes.
To learn about storing artifacts and debug samples, see Storage.
ClearML supports multiple Task types for different workflows, including:
training (default), testing, inference
monitor, service, application
Your own implementations for Tasks
Task states and state transitions¶
The state of a Task represents its stage in the Task lifecycle. It indicates whether the Task is read-write (editable) or read-only. For each state, a state transition indicates which actions can be performed on an experiment and the new state after performing an action.
The following table describes Task the states and state transitions.
|State||Description / Usage||State Transition|
|Draft||The experiment is editable. Only experiments whose status is Draft are editable. The experiment is not running locally or remotely.||If the experiment is enqueued for a worker to fetch and execute, the state becomes Pending.|
|Pending||The experiment was enqueued. It is in a queue waiting for a worker to fetch and execute it.||If the experiment is dequeued, the state becomes Draft.|
|Running||The experiment is running locally or remotely.||If the experiment is manually or programmatically terminated, the state becomes Aborted.|
|Completed||The experiment ran and terminated successfully.||If the experiment is reset or cloned, the state of cloned experiment or newly cloned experiment becomes Draft. Resetting deletes the logs and output of a previous run. Cloning creates an exact, editable copy.|
|Failed||The experiment ran and terminated with an error.||The same as Completed.|
|Aborted||The experiment ran, and was manually or programmatically terminated.||The same as Completed.|
|Published||The experiment is read-only. Publish an experiment to prevent changes to its inputs and outputs.||A Published experiment cannot be reset. If it is cloned, the state of the newly cloned experiment becomes Draft.|