Tasks Examples

Trains Task types

By default, calling the Task.init method creates a training Task. Trains supports multiple Task types for different workflows, including:

  • Task.TaskTypes.training (Default)
  • Task.TaskTypes.testing
  • Task.TaskTypes.application
  • Task.TaskTypes.controller
  • Task.TaskTypes.data_processing
  • Task.TaskTypes.inference
  • Task.TaskTypes.monitor
  • Task.TaskTypes.optimizer
  • Task.TaskTypes.qc
  • Task.TaskTypes.service
  • Task.TaskTypes.custom

To create a Task (experiment) whose type is not training, call the Task.init method, and specify the task_type parameter.

For example, create a testing Task.

task = Task.init(project_name='User Guide Documentation',task_name='Task example',
    task_type=Task.TaskTypes.testing)

Forcing a new Task

When running your Task code in a development environment for the first time, Trains creates a new Task and assigns it a new Task Id.

If the same code runs again, and the previous Task is not published (see Experiment states), Trains overwrites the experiment's previous output data and any changes you may have made to the Task in code or in the Trains Web-App (UI).

If the previous Task was published, Trains creates a new Task and assigns it a new Task Id. In this case, you see another Task in the Trains Web-App (UI), in the same project and with the same experiment name.

However, when running your Task code in a development environment, you can force Trains to create a new Task and assign it a new Task ID each time your code runs.

To force Trains to create a new Task, call the Task.init method, and set the reuse_last_task_id parameter argument to False.

Task (experiments) running remotely

When running a Task remotely, (meaning, enqueuing a Task to run which is already in Trains, or running a Task directly using Trains Agent), Trains does not create a new Task even if reuse_last_task_id=False was specified in the code.

task = Task.init('User Guide Documentation','Task example force new',reuse_last_task_id=False)

argparse parameters

Trains automatically tracks argparse parameters. You can also disable this feature and, optionally, connect your own parameter dictionary.

For example, if you use argparse, Trains automatically tracks the argparse object's arguments.

task = Task.init('User Guide Documentation','Task example arg parse')

parser = argparse.ArgumentParser(description='PyTorch MNIST Example')
parser.add_argument('--batch-size', type=int, default=64)
parser.add_argument('--test-batch-size', type=int, default=1000)
parser.add_argument('--epochs', type=int, default=2)
args = parser.parse_args()

In the Trains Web-App (UI), view the hyperparameters. You can see that Trains is tracking the argparse object's arguments.

If you do not want Trains to automatically track the argparse object's arguments, when calling the Task.init method, set the auto_connect_arg_parser parameter argument to False.

task = Task.init('User Guide Documentation','Task example no arg parse', 
    auto_connect_arg_parser=False)

parser = argparse.ArgumentParser(description='PyTorch MNIST Example')
parser.add_argument('--batch-size', type=int, default=64)
parser.add_argument('--test-batch-size', type=int, default=1000)
parser.add_argument('--epochs', type=int, default=2)
args = parser.parse_args()

dict = {'param_A': 123, 'param_B': 456, 'param_C': 'some string'}

task.connect(dict)

In the Trains Web-App (UI), view the hyperparameters. You can see that Trains is tracking the parameter dictionary's parameters, not the argparse object's arguments.

Task pipelining

A Task pipeline automates a workflow. In Trains, a Task pipeline is a series of Tasks, where each Task can perform some operations, clone other Tasks, optionally pass the other Tasks some parameters, and enqueue the cloned Tasks to run.

In this example, we demonstrate a Task pipeline composed of two Tasks. One Task is a controller Task which manages the pipeline. The second Task is a base Task representing any Task doing work in a pipeline. The controller Task creates hyperparameters, clones the base Task, passes hyperparameters to the clone of the base Task, and then enqueues the clone of the base Task to run.

Since the controller Task clones the base Task, the base Task code must run at least once before the controller Task, so that the base Task is already in the Trains backend when the controller Task tries to clone it.

We use two scripts which are in trains repository:

Step 1. The base Task

Create the base Task Toy Base Task by running the toy_base_task.py script. Do this first, because the base Task must be in the Trains backend before it is cloned in Step 3.

Toy Base Task is simple. It creates a parameter dictionary containing a parameter named Example_Param, connects the parameter dictionary to the Task, and then prints the parameter.

You can see that the parameter Example_Param is assigned the value 1.

# Initialize the task pipe's first task used to start the task pipe
task = Task.init('examples', 'Toy Base Task')

# Create a dictionary for hyper-parameters
params = {}

# Add a parameter and value to the dictionary
params['Example_Param'] = 1

# Connect the hyper-parameter dictionary to the task
task.connect(params)

# Print the value to demonstrate it is the value is set by the initiating task.
print ("Example_Param is", params['Example_Param'])

Step 2. The controller Task

Run the controller Task Simple Controller Task by running the script task_piping_example.py.

The controller Task creates a parameter dictionary containing the following parameters:

  • next_task_name - The name of the next Task.
  • param_name - The same parameter used in the base Task: Example_Param
  • param_name_new_value - A new value for the parameter used in the base Task.
  • execution_queue_name - The name of the queue for enqueuing the cloned base Task.

You can see that in Simple Controller Task, the parameter Example_Param is assigned the value 3.

# Initialize the Task Pipe's first Task used to start the Task Pipe
task = Task.init('examples', 'Simple Controller Task')

# Create a hyper-parameter dictionary for the task
param = {}
# Connect the hyper-parameter dictionary to the task
param = task.connect(param)

# In this example we pass next task's name as a parameter
param['next_task_name'] = 'Toy Base Task'
# This is a parameter name in the next task we want to change
param['param_name'] = 'Example_Param'
# This is the parameter value in the next task we want to change
param['param_name_new_value'] = 3
# The queue where we want the template task (clone) to be sent to
param['execution_queue_name'] = 'default'

Simple Controller Task performs some work. This represents whatever work a Task in a pipeline may do.

# Simulate the work of a Task
print('Processing....')
sleep(2.0)
print('Done processing :)')

Get the base Task and clone it. The name of the cloned base Task is Auto generated cloned task.

# Get a reference to the task to pipe to.
next_task = Task.get_task(project_name=task.get_project_name(), task_name=param['next_task_name'])

# Clone the task to pipe to. This creates a task with status Draft whose parameters can be modified.
cloned_task = Task.clone(source_task=next_task, name='Auto generated cloned task')

Get the parameters from the cloned base Task, and change the value of the cloned base Task's parameter Example_Param from 1 to 3.

# Get the original parameters of the Task, modify the value of one parameter,
#   and set the parameters in the next Task
cloned_task_parameters = cloned_task.get_parameters()
cloned_task_parameters[param['param_name']] = param['param_name_new_value']
cloned_task.set_parameters(cloned_task_parameters)

Enqueue the cloned base Task Auto generated cloned task to run.

# Enqueue the Task for execution. The enqueued Task must already exist in the trains platform
print('Enqueue next step in pipeline to queue: {}'.format(param['execution_queue_name']))
Task.enqueue(cloned_task.id, queue_name=param['execution_queue_name'])

Step 3. Running the controller Task

When you run Simple Controller Task by running the script task_piping_example.py, you can view the cloned base Task Auto generated cloned task in the Trains Web-App (UI).

The Example_Param has a value of 3. This is the value assigned by the controller Task.