Trains Agent Reference

This reference page provides details information about Trains Agent commands, including:

  • build - Create a worker environment, without executing an experiment.
  • config - List your Trains Agent configuration.
  • daemon - Run a worker daemon listening to a queue for Task (experiments) to execute.
  • execute - Execute an experiment, locally without a queue.
  • list - List the current workers.

build

Use the build subcommand to create a worker environment, but do not execute the experiment.

Syntax

trains-agent build [-h] --id TASK_ID [--target TARGET]
                          [--install-globally]
                          [--docker [DOCKER [DOCKER ...]]]
                          [--python-version PYTHON_VERSION]
                          [--entry-point {reuse_task,clone_task}] [-O]
                          [--git-user GIT_USER] [--git-pass GIT_PASS]
                          [--log-level {DEBUG,INFO,WARN,WARNING,ERROR,CRITICAL}]
                          [--gpus GPUS] [--cpu-only]

Arguments

Argument Description Mandatory
id Build a worker environment for this Task ID. Yes
cpu-only Disable GPU access for the Docker container. No
docker Docker mode. A Docker container that a worker will execute at launch.

To specify the image name and optional arguments, either:

  • use --docker <image_name> <args> on the command line, or
  • use --docker on the command line, and specify the image name and arguments in the configuration file, see the agent section on the "Trains Agent Reference" page.

Environment variable settings for Dockers:

  • TRAINS_DOCKER_SKIP_GPUS_FLAG - Ignore the gpus flag inside the Docker container. This also allows you to execute Trains Agent using Docker versions earlier than 19.03.
  • NVIDIA_VISIBLE_DEVICES - Limit GPU visibility for the Docker container.
  • TRAINS_AGENT_GIT_USER and TRAINS_AGENT_GIT_PASS - Pass these credentials to the Docker container at execution.

To limit GPU visibility for Docker, set the NVIDIA_VISIBLE_DEVICES environment variable.

No
entry-point Used in conjunction with --docker, indicates how to run the Task specified by task-id on Docker startup.

The entry-point options are:

  • reuse - Overwrite the existing Task data.
  • clone_task - Clone the Task, and execute the cloned Task.
No
git-pass Git password for repository access. No
git-user Git username for repository access. No
gpus Specify the active GPUs for the Docker containers to use. These are the same GPUs set in the NVIDIA_VISIBLE_DEVICES environment variable.

For example:

  • --gpus 0
  • --gpu 0,1,2
  • --gpus all
No
h, help Get help for this command. No
install-globally Install the required Python packages before creating the virtual environment. Use agent.package_manager.system_site_packages to control the installation of the system packages. When --docker is used, install-globally is always true. No
log-level SDK log level.

The values are:

  • DEBUG
  • INFO
  • WARN
  • WARNING
  • ERROR
  • CRITICAL
No
python-version Virtual environment Python version to use. No
O Compile optimized pyc code (see python documentation). Repeat for more optimization. No
target The target folder for the virtual environment and source code that will be used at launch. No

config

Use the config subcommand to list your Trains Agent configuration.

Syntax

trains-agent config

daemon

Use the daemon subcommand to run a worker, optionally in a Docker container, listening to a queue.

Syntax

trains-agent daemon [-h] [--foreground] [--queue QUEUES [QUEUES ...]]
                           [--standalone-mode] [--services-mode]
                           [--create-queue] [--detached] [-O]
                           [--git-user GIT_USER] [--git-pass GIT_PASS]
                           [--log-level {DEBUG,INFO,WARN,WARNING,ERROR,CRITICAL}]
                           [--gpus GPUS] [--cpu-only]
                           [--docker [DOCKER [DOCKER ...]]]
                           [--force-current-version]

Arguments

Argument Description Mandatory
cpu-only If running in Docker mode (see the docker option), disable GPU access for the Docker container. No
create-queue If the queue name provided does not exist in the server, create and use it. No
detach Run in the background. The trains-agent command returns immediately. No
docker Run in Docker mode. Execute the Task inside a Docker container.

To specify the image name and optional arguments, either:

  • use --docker <image_name> <args> on the command line, or
  • use --docker on the command line, and specify the image name and arguments in the configuration file, see the agent section on the "Trains Agent Reference" page.

Environment variable settings for Dockers:

  • TRAINS_DOCKER_SKIP_GPUS_FLAG - Ignore the gpus flag inside the Docker container. This also allows you to execute Trains Agent using Docker versions earlier than 19.03.
  • NVIDIA_VISIBLE_DEVICES - Limit GPU visibility for the Docker container.
  • TRAINS_AGENT_GIT_USER and TRAINS_AGENT_GIT_PASS - Pass these credentials to the Docker container at execution.

No
force-current-version To use your current version of Trains Agent when running in Docker mode (the docker argument is specified), instead of the latest Trains Agent version which is automatically installed, specify force-current-version.

For example, if your current Trains Agent version is 0.13.1, but the latest version of Trains Agent is 0.13.3, use --force-current-version and your Task will execute in the Docker container with Trains Agent version 0.13.1. 		No
foreground Pipe full log to stdout/stderr. Do not use this option if running in background. No
git-pass Git password for repository access. No
git-user Git username for repository access. No
gpus If running in Docker mode (see the docker option), specify the active GPUs for the Docker containers to use.

These are the same GPUs set in the NVIDIA_VISIBLE_DEVICES environment variable.

For example:

  • --gpus 0
  • --gpu 0,1,2
  • --gpus all
No
log-level SDK log level.

The values are:

  • DEBUG
  • INFO
  • WARN
  • WARNING
  • ERROR
  • CRITICAL
No
O Compile optimized pyc code (see python documentation). Repeat for more optimization. No
queue Specify the queues which the worker is listening to. The values can be any combination of:
  • One or more queue Ids.
  • One or more queue names.
  • default indicating the default queue.
No
services-mode Spin multiple, simultaneous Tasks, each in its own Docker container, on the same machine. Each Task will be registered as a new node in the system, providing tracking and transparency capabilities. Start up and shutdown of each Docker is verified. Use in CPU mode (--cpu-only), only. No
standalone-mode Do not use any network connects. This assumes all requirements are pre-installed. No

execute

Use the execute subcommand to build and execute an experiment without a queue.

Syntax

trains-agent execute [-h] --id TASK_ID [--log-file LOG_FILE]
                            [--disable-monitoring] [--full-monitoring]
                            [--require-queue] [--standalone-mode]
                            [--docker [DOCKER [DOCKER ...]]] [--clone] [-O]
                            [--git-user GIT_USER] [--git-pass GIT_PASS]
                            [--log-level {DEBUG,INFO,WARN,WARNING,ERROR,CRITICAL}]

Arguments

Argument Description Mandatory
id The ID of the Task to build. Yes
clone Clone the Task specified by id, and then execute that cloned Task. No
docker Run in Docker mode. Execute the Task inside a Docker container.

To specify the image name and optional arguments, either:

  • use --docker <image_name> <args> on the command line, or
  • use --docker on the command line, and specify the default image name and arguments in the configuration file, see the agent section on the "Trains Agent Reference" page.

Environment variable settings for Dockers containers:

  • TRAINS_DOCKER_SKIP_GPUS_FLAG - Ignore the gpus flag inside the Docker container. This also allows you to execute Trains Agent using Docker versions earlier than 19.03.
  • NVIDIA_VISIBLE_DEVICES - Limit GPU visibility for the Docker container.
  • TRAINS_AGENT_GIT_USER and TRAINS_AGENT_GIT_PASS - Pass these credentials to the Docker container at execution.

No
disable-monitoring Disable logging and monitoring, except for stdout. No
full-monitoring Create a full log, including the environment setup log, Task log, and monitoring, as well as stdout. No
git-pass Git password for repository access. No
git-user Git username for repository access. No
h, help Get help for this command. No
log-file The log file for Task execution output (stdout / stderr) to a text file. No
log-level SDK log level.

The values are:

  • DEBUG
  • INFO
  • WARN
  • WARNING
  • ERROR
  • CRITICAL
No
O Compile optimized pyc code (see python documentation). Repeat for more optimization. No

list

Use the list subcommand to list information about all workers

Syntax

trains-agent list