Trains Agent Reference

This reference page provides details information about Trains Agent commands and use cases, including the following trains-agent command, sub-commands:

  • build - Create a worker environment, without executing an experiment.
  • config - List your Trains Agent configuration.
  • daemon - Run a worker daemon listening to a queue for Task (experiments) to execute.
  • execute - Execute an experiment, locally without a queue.
  • list - List the current workers.

build

Use the build subcommand to create a worker environment, but do not execute the experiment.

Syntax

trains-agent build [-h] --id TASK_ID [--target TARGET]
                          [--docker [DOCKER [DOCKER ...]]] [--gpus GPUS]
                          [--cpu-only] [--python-version PYTHON_VERSION] [-O]
                          [--git-user GIT_USER] [--git-pass GIT_PASS]
                          [--log-level {DEBUG,INFO,WARN,WARNING,ERROR,CRITICAL}]

Arguments

Argument Description Mandatory
id The Id of the Task to build. Yes
cpu-only If running in Docker mode (see the docker option), disable GPU access for the Docker container. No
docker Run in Docker mode. Execute the Task inside a Docker container.

To specify the image name and optional arguments, either:

  • use --docker <image_name> <args> on the command line, or
  • use --docker on the command line, and specify the image name and arguments in the configuration file, see the agent section on the "Trains Agent Reference" page.

To ignore the gpus flag inside the Docker container, set the TRAINS_DOCKER_SKIP_GPUS_FLAG environment variable. This also allows you to execute Trains Agent using Docker versions earlier than 19.03.

To limit GPU visibility for Docker, set the NVIDIA_VISIBLE_DEVICES environment variable.

No
git-pass Git password for repository access. No
git-user Git username for repository access. No
gpus If running in Docker mode (see the docker option), specify the active GPUs for the Docker containers to use. These are the same GPUs set in the NVIDIA_VISIBLE_DEVICES environment variable.

For example:

  • --gpus 0
  • --gpu 0,1,2
  • --gpus all

No
h, help Get help for this command. No
log-level SDK log level. The values are:
  • DEBUG
  • INFO
  • WARN
  • WARNING
  • ERROR
  • CRITICAL
No
python-version Virtual environment Python version to use. No
O Compile optimized pyc code (see python documentation). Repeat for more optimization. No
target The target folder for the Task's virtual environment and source code. No

config

Use the config subcommand to list your Trains Agent configuration.

Syntax

trains-agent config

daemon

Use the daemon subcommand to run a worker, optionally in a Docker container, listening to a queue.

Syntax

trains-agent daemon [-h] [--foreground] [--queue QUEUES [QUEUES ...]]
                           [--standalone-mode] [--detached] [-O]
                           [--git-user GIT_USER] [--git-pass GIT_PASS]
                           [--log-level {DEBUG,INFO,WARN,WARNING,ERROR,CRITICAL}]
                           [--docker [DOCKER [DOCKER ...]]] [--gpus GPUS]
                           [--cpu-only] [--force-current-version]

Arguments

Argument Description Mandatory
cpu-only If running in Docker mode (see the docker option), disable GPU access for the Docker container. No
detach Run in the background. The trains-agent command returns immediately. No
docker Run in Docker mode. Execute the Task inside a Docker container.

To specify the image name and optional arguments, either:

  • use --docker <image_name> <args> on the command line, or
  • use --docker on the command line, and specify the image name and arguments in the configuration file, see the agent section on the "Trains Agent Reference" page.

To ignore the gpus flag inside the Docker container, set the TRAINS_DOCKER_SKIP_GPUS_FLAG environment variable. This also allows you to execute Trains Agent using Docker versions earlier than 19.03.

To limit GPU visibility for Docker, set the NVIDIA_VISIBLE_DEVICES environment variable.

No
force-current-version To use your current version of Trains Agent when running in Docker mode (the docker argument is specified), instead of the latest Trains Agent version which is automatically installed, specify force-current-version.

For example, if your current Trains Agent version is 0.13.1, but the latest version of Trains Agent is 0.13.3, use --force-current-version and your Task will execute in the Docker container with Trains Agent version 0.13.1.
No
foreground Pipe full log to stdout/stderr. Do not use this option if running in background. No
git-pass Git password for repository access. No
git-user Git username for repository access. No
gpus If running in Docker mode (see the docker option), specify the active GPUs for the Docker containers to use.

These are the same GPUs set in the NVIDIA_VISIBLE_DEVICES environment variable.

For example:

  • --gpus 0
  • --gpu 0,1,2
  • --gpus all

No
log-level SDK log level. The values are:
  • DEBUG
  • INFO
  • WARN
  • WARNING
  • ERROR
  • CRITICAL
No
O Compile optimized pyc code (see python documentation). Repeat for more optimization. No
queue Specify the queues which the worker is listening to. The values can be any combination of:
  • One or more queue Ids.
  • One or more queue names.
  • default indicating the default queue.
No
standalone-mode Do not use any network connects. This assumes all requirements are pre-installed. No

execute

Use the execute subcommand to build and execute an experiment without a queue.

Syntax

trains-agent execute [-h] --id TASK_ID [--log-file LOG_FILE]
                            [--disable-monitoring] [--full-monitoring]
                            [--require-queue] [--standalone-mode]
                            [--docker [DOCKER [DOCKER ...]]] [--clone] [-O]
                            [--git-user GIT_USER] [--git-pass GIT_PASS]
                            [--log-level {DEBUG,INFO,WARN,WARNING,ERROR,CRITICAL}]

Arguments

Argument Description Mandatory
id The Id of the Task to build. Yes
clone Clone the Task specified by id, and then execute that cloned Task. No
docker Run in Docker mode. Execute the Task inside a Docker container.

To specify the image name and optional arguments, either:

  • use --docker <image_name> <args> on the command line, or
  • use --docker on the command line, and specify the default image name and arguments in the configuration file, see the agent section on the "Trains Agent Reference" page.

To ignore the gpus flag inside the Docker container, set the TRAINS_DOCKER_SKIP_GPUS_FLAG environment variable. This also allows you to execute Trains Agent using Docker versions earlier than 19.03.

To limit GPU visibility for Docker, set the NVIDIA_VISIBLE_DEVICES environment variable.

No
disable-monitoring Disable logging and monitoring, except for stdout. No
full-monitoring Create a full log, including the environment setup log, Task log, and monitoring, as well as stdout. No
git-pass Git password for repository access. No
git-user Git username for repository access. No
h, help Get help for this command. No
log-file The log file for Task execution output (stdout / stderr) to a text file. No
log-level SDK log level. The values are:
  • DEBUG
  • INFO
  • WARN
  • WARNING
  • ERROR
  • CRITICAL
No
O Compile optimized pyc code (see python documentation). Repeat for more optimization. No

list

Use the list subcommand to list information about all workers

Syntax

trains-agent list

Use Case Examples

Running worker daemons with queues

The default queue

  • Run a worker daemon listening to the default queue to fetch Tasks and execute them:
    trains-agent daemon --queue default

User-created queues

  • Run a worker daemon listening to user-created queued to fetch Tasks and execute them:
    trains-agent daemon --queue <your_queue>

Prioritizing queues

  • Prioritize queues by specifying more than one queue after the queue option. The order of queues in the command determines the priorities of the queues (first queue is the highest priority).
    trains-agent daemon --queue important_jobs default
    The Trains Agent will first try to pull jobs from the important_jobs queue, only then it will fetch a job from the default queue.

Docker mode

  • Run a worker daemon in a Docker container:
    trains-agent daemon --queue default --docker
  • Use your current Trains Agent version in the Docker container to execute your Task, instead of the latest Trains Agent version which is automatically installed.
    trains-agent daemon --queue default --docker --force-current-version
  • Run a worker daemon in a Docker container ignoring the gpus flag (also run a worker daemon in Docker version earlier than version 19.03).

    Set the environment variable TRAINS_DOCKER_SKIP_GPUS_FLAG = true.

  • For Kubernetes, specify a host mount on your daemon host. Do not use the host mount inside the Docker container.

    Set the environment variable TRAINS_AGENT_K8S_HOST_MOUNT.

    For example:

    TRAINS_AGENT_K8S_HOST_MOUNT=/mnt/host/data:/root/.trains

Specifying GPUs

  • Run a worker daemon with more than one GPU on the same machine:
    trains-agent daemon --gpus 0 --queue default &
    trains-agent daemon --gpus 1 --queue default &
  • Run a worker daemon listening with more than one GPU and a dedicated queue:
    trains-agent daemon --gpus 0,1 --queue dual_gpu &
    trains-agent daemon --gpus 2,3 --queue dual_gpu &

Debugging

  • Running Trains Agent in foreground mode sending all output to the console:
    trains-agent daemon --queue default --foreground

Executing Tasks without queues

Execute a Task without queue

  • Execute a Task in a worker without a queue:
    trains-agent execute --id <task-id>

Clone a Task and execute the cloned Task

  • Clone the specified Task and execute the cloned Task in worker without a queue:
    trains-agent execute --id <task-id> --clone

Docker mode

  • Execute a Task in worker using a Docker container without a queue:
    trains-agent execute --id <task-id> --docker