Trains AWS Autoscaler Service

Trains is now ClearML

This documentation applies to the legacy Trains versions. For the latest documentation, see ClearML.

The Trains AWS autoscaler optimizes AWS EC2 instance scaling according to the instance types you want to use, and the budget you configure. In the budget, you set the maximum number of each instance type to spin up for experiments awaiting execution in a specific queue. Configure multiple instance types per queue, and multiple queues. The Trains AWS autoscaler will spin down idle instances based on a maximum idle time and polling interval you configure. Its Task name is AWS Auto-Scaler and it is associated with the DevOps project.

The Trains AWS autoscaler can execute in the Trains Agent services container. The Trains AWS autoscaler is configurable. It is pre-loaded in Trains Server and its status is Draft (editable). You can set the instance types and configure your budget in the Trains Web (UI), and then enqueue the Task to the services queues. Or, run the script, with options to run locally or as a service.

Running the Trains AWS autoscaler

You can run the Trains AWS autoscaler two ways:

Running using the Trains Web (UI)

Edit the parameters for the instance types, edit budget configuration by editing the Task, and then enqueue the Task to run in Trains Agent services mode.

  1. In the Trains Web (UI) > Projects page > DevOps project > AWS Auto-Scaler Task.
  2. Set the AWS and Git credentials, parameters for idle AWS EC2 instances, and a worker prefix.

    • In the CONFIGURATIONS tab > HYPER PARAMETERS > Args > hover > EDIT.
      • cloud_credentials_key - AWS access key.
      • cloud_credentials_region - AWS region.
      • cloud_credentials_secret - AWS access secret.
      • cloud_provider - AWS.
      • default_docker_image - The default Docker image to use for the AWS EC2 instance.
      • git_pass - Git password.
      • git_user - Git username.
      • max_idle_time_min - The maximum time an AWS EC2 instance can be idle before the Trains AWS autoscaler spins it down.
      • polling_interval_time_min - How often the Trains AWS autoscaler checks for idle instances.
      • workers_prefix
    View a screenshot


  3. Configure the budget.

    • In CONFIGURATION OBJECTS > General, hover > EDIT. Edit the resource_configurations dictionary:

      resource_configurations {
          <resource-name> {
            instance_type = "<instance_type>"
            is_spot = <boolean>
            availability_zone = "<AWS-region>"
            ami_id = "<AMI-ID>"
            ebs_device_name = "<EBS-device-name>"
            ebs_volume_size = <EBS-size-in-GB>
            ebs_volume_type = "<EBS-vol-type>"
      queues {
          <queue-name> = [["<resource-name>", <max-instances-of-resource-name>]]
      extra_trains_conf = "<Trains-config-file>"
      extra_vm_bash_script = "<bash-script>"


      • <resource-name> - The name you assign to each resource (AWS EC2 instance type). Used in the budget.
      • queues - The Trains AWS autoscaler will optimize scaling for experiments awaiting execution in these queues.
      • <queue-name> - A specific queue.
      • <max-instances-of-resource-name> - The maximum number of instances of the specified resource-name to spin up.
      • is_spot - If true, then use a spot instance. If false, then use a reserved instance.
      • extra_trains_conf - A Trains configuration file to use for executing experiments in Trains Agent.
      • extra_vm_bash_script - A bash script to execute when creating an instance, before Trains Agent executes.
    View a screenshot


  4. Set the Task to run in Trains Agent services mode.

    1. In HYPER PARAMETERS > Args > hover > EDIT.

    2. Change the remote parameter to true.

    View a screenshot


  5. Click SAVE.

  6. In the experiments table, right click the AWS Auto-Scaler Task > Enqueue > services queue > ENQUEUE.

Running using the script

The script includes a wizard which prompts you for the instance details, and budget configuration.

The script can run in two ways:

  • Configure and enqueue.
  • Enqueue with an existing configuration.

To configure and enqueue:

Use the run command line option:

python --run

When the script runs, a configuration wizard prompts you for all required information.

View the configuration wizard steps
  1. The setup wizard begins. Enter the AWS credentials and AWS region name.

    AWS Autoscaler setup wizard
    Follow the wizard to configure your AWS auto-scaler service.
    Once completed, you will be able to view and change the configuration in the trains-server web UI.
    It means there is no need to worry about typos or mistakes :)
    Enter AWS Access Key ID : 
    Enter AWS Secret Access Key : 
    Enter AWS region name [us-east-1b]:
  2. Enter Git credentials. These are required by Trains Agent to setup a Task execution environment in an AWS EC2 instance.

    GIT credentials:
    Enter GIT username for repository cloning (leave blank for SSH key authentication): []
    Enter password for user '<username>':

    The wizard reports the Git credentials it will use.

    Git repository cloning will be using user=*************** password=***********
  3. Enter the default Docker image and parameters to use.

    Enter default docker image/parameters to use [nvidia/cuda:10.1-runtime-ubuntu18.04]:
  4. For each AWS EC2 instance type you will use in your budget, choose the instance tyoe, whether to use spot instances, select an AMI, and define the Amazon EBS volume. Select as many instance types as you may need.

    Configure the machine types for the auto-scaler:
    Select Amazon instance type ['g4dn.4xlarge']:
    Use spot instances? [y/N]: y
    Select availability zone ['us-east-1b']:
    Select the Amazon Machine Image id ['ami-07c95cafbb788face']:
    Enter the Amazon EBS device ['/dev/xvda']:
    Enter the Amazon EBS volume size (in GiB) [100]:
    Enter the Amazon EBS volume type ['gp2']:

    Name the instance type you configured. Later in the configuration, you use this name to create your budget.

    Select a name for this instance type (used in the budget section) For example 'aws4gpu':

    The wizard prompts to select another instance type.

    Define another instance type? [y/N]:
  5. Enter any bash script you want to run on newly created instances, before Trains Agent executes.

    Enter any pre-execution bash script to be executed on the newly created instances []:
  6. Configure the AWS autoscaler budget. For each queue that you want to use in your budget, select the queue, and the maximum number of each instance type that the Trains AWS autoscaler can spin up to execute experiments awaiting execution in that queue.

    Define the machines budget:
    Select a queue name (for example: 'aws_4gpu_machines') : 
    Select a instance type to attach to the queue ['aws-g4dn.xlarge', 'aws-g4dn.8xlarge', 'aws-g4dn.16xlarge']:
    Enter maximum number of 'aws-g4dn.xlarge' instances to spin simultaneously (example: 3) :
  7. If you want to add another instance type to the same queue, add it. The previous step repeats.

    Do you wish to add another instance type to queue? [y/N]:
  8. The Trains AWS autoscalar polls instances, and if they have been idle for the maximum idle time you specify, the autoscaler spins them down.

    Enter maximum idle time for the auto-scaler to spin down an instance (in minutes) [15]:
    Enter instances polling interval for the auto-scaler (in minutes) [5]:

    The configuration is complete. Trains initializes the Task AWS Auto-Scaler, the service begins, and the script prints a hyperlink to the Task's log.

    TRAINS Task: created new task id=d0ee5309a9a3471d8802f2561da60dfa
    TRAINS Monitor: GPU monitoring failed getting GPU reading, switching off GPU monitoring
    TRAINS results page:
    Running AWS auto-scaler as a service
    Execution log

To enqueue with an existing configuration:

Use the remote command line option:

python --remote

When the script runs, it allows you to create a new configuration.