Trains Agent Reference

This page provides a reference for the Trains Agent commands and use cases, including the following trains-agent command, sub-commands:

  • build - Create a worker environment, without executing an experiment.
  • config - List your Trains Agent configuration.
  • daemon - Run a worker daemon listening to a queue for Task (experiments) to execute.
  • execute - Execute an experiment, locally without a queue.
  • list - List the current workers.

build

Use the build subcommand to create a worker environment without an experiment to run in it.

Command syntax

trains-agent build [-h] --id TASK_ID [--target-folder TARGET_FOLDER]
                          [--python-version PYTHON_VERSION] [-O]
                          [--git-user GIT_USER] [--git-pass GIT_PASS]
                          [--log-level {DEBUG,INFO,WARN,WARNING,ERROR,CRITICAL}]

Command arguments

Argument Description Mandatory
id The Id of the Task to build. Yes
git-pass Git password for repository access. No
git-user Git username for repository access. No
h, help Get help for this command. No
log-level SDK log level. The values are:
  • DEBUG
  • INFO
  • WARN
  • WARNING
  • ERROR
  • CRITICAL
No
python-version Virtual environment Python version to use. No
O Compile optimized pyc code (see python documentation). Repeat for more optimization. No
target-folder The target folder for the Task's virtual environment and source code. No

config

Use the config subcommand to list your Trains Agent configuration.

Command syntax

trains-agent config

daemon

Use the daemon subcommand to run a worker, optionally in a Docker container, listening to a queue.

Command syntax

trains-agent daemon [-h] [--foreground]
                           [--docker [DOCKER [DOCKER ...]]]
                           [--queue QUEUES [QUEUES ...]] [-O]
                           [--git-user GIT_USER] [--git-pass GIT_PASS]
                           [--log-level {DEBUG,INFO,WARN,WARNING,ERROR,CRITICAL}]

Command arguments

Argument Description Mandatory
docker Run the execution Task inside a Docker container (v19.03 and above). The Docker image and optional arguments are specified in either:
  • Command after the docker option (e.g. --docker <image_name> <args>)
  • The Trains configuration file, agent.default_docker.image and agent.default_docker.arguments options.
Set the environment variable NVIDIA_VISIBLE_DEVICES to limit gpu visibility for Docker. 		No
foreground Pipe full log to stdout/stderr. Do not use this option if running in background. No
git-pass Git password for repository access. No
git-user Git username for repository access. No
log-level SDK log level. The values are:
  • DEBUG
  • INFO
  • WARN
  • WARNING
  • ERROR
  • CRITICAL
No
O Compile optimized pyc code (see python documentation). Repeat for more optimization. No
queue Specify the queues which the worker is listening to. The values can be any combination of:
  • One or more queue Ids.
  • One or more queue names.
  • default indicating the default queue.
No

execute

Use the execute subcommand to build and execute an experiment without a queue.

Command syntax

trains-agent execute [-h] --id TASK_ID [--log-file LOG_FILE]
                            [--disable-monitoring] [--full-monitoring] [-O]
                            [--git-user GIT_USER] [--git-pass GIT_PASS]
                            [--log-level {DEBUG,INFO,WARN,WARNING,ERROR,CRITICAL}]

Command arguments

Argument Description Mandatory
id The Id of the Task to build. Yes
disable-monitoring Disable logging and monitoring, except for stdout. No
full-monitoring Create a full log, including the environment setup log, Task log, and monitoring, as well as stdout. No
git-pass Git password for repository access. No
git-user Git username for repository access. No
h, help Get help for this command. No
log-file The log file for Task execution output (stdout / stderr) to a text file. No
log-level SDK log level. The values are:
  • DEBUG
  • INFO
  • WARN
  • WARNING
  • ERROR
  • CRITICAL
No
python-version Virtual environment Python version to use. No
O Compile optimized pyc code (see python documentation). Repeat for more optimization. No
target-folder The target folder for the Task's virtual environment and source code. No

list

Use the list subcommand to list information about all workers

Command syntax

trains-agent list

Use Cases Examples

Default queue

Run a worker daemon listening to the default queue:

trains-agent daemon --queue default

User-created queues

Run a worker daemon listening to user-created queued:

trains-agent daemon --queue <your_queue>

For example:

trains-agent daemon --queue MyQueue

Specifying GPUs

Run a worker daemon with more than one GPU on the same machine:

trains-agent daemon --gpus 0 --queue default &
trains-agent daemon --gpus 1 --queue default &

Run a worker daemon listening with more than one GPU and a dedicated queue:

trains-agent daemon --gpus 0,1 --queue dual_gpu &
trains-agent daemon --gpus 2,3 --queue dual_gpu &

Docker mode

Run a worker daemon in a Docker container:

trains-agent daemon --queue default --docker

Prioritizing queues

Prioritize queues by specifying more than one queue after the queue option. The order of queues in the command determines the priorities of the queues (first queue is highest priority).

trains-agent daemon --queue important_jobs default

The Trains Agent will first try to pull jobs from the important_jobs queue, only then it will fetch a job from the default queue.

Debugging

Running Trains Agent in foreground mode sending all output to the console:

trains-agent daemon --queue default --foreground