Trains Agent Reference

This reference page provides details information about Trains Agent commands and use cases, including the following trains-agent command, sub-commands:

build - Create a worker environment, without executing an experiment.
config - List your Trains Agent configuration.
daemon - Run a worker daemon listening to a queue for Task (experiments) to execute.
execute - Execute an experiment, locally without a queue.
list - List the current workers.

build

Use the build subcommand to create a worker environment, but do not execute the experiment.

Syntax

trains-agent build [-h] --id TASK_ID [--target TARGET]
                          [--docker [DOCKER [DOCKER ...]]] [--gpus GPUS]
                          [--cpu-only] [--python-version PYTHON_VERSION] [-O]
                          [--git-user GIT_USER] [--git-pass GIT_PASS]
                          [--log-level {DEBUG,INFO,WARN,WARNING,ERROR,CRITICAL}]

Arguments

Argument	Description	Mandatory
id	The Id of the Task to build.	Yes
cpu-only	If running in Docker mode (see the docker option), disable GPU access for the Docker container.	No
docker	Run in Docker mode. Execute the Task inside a Docker container. To specify the image name and optional arguments, either: use --docker <image_name> <args> on the command line, or use --docker on the command line, and specify the image name and arguments in the configuration file, see the agent section on the "Trains Agent Reference" page. To ignore the gpus flag inside the Docker container, set the TRAINS_DOCKER_SKIP_GPUS_FLAG environment variable. This also allows you to execute Trains Agent using Docker versions earlier than 19.03. To limit GPU visibility for Docker, set the NVIDIA_VISIBLE_DEVICES environment variable.	No
git-pass	Git password for repository access.	No
git-user	Git username for repository access.	No
gpus	If running in Docker mode (see the docker option), specify the active GPUs for the Docker containers to use. These are the same GPUs set in the NVIDIA_VISIBLE_DEVICES environment variable. For example: --gpus 0 --gpu 0,1,2 --gpus all	No
h, help	Get help for this command.	No
log-level	SDK log level. The values are: DEBUG INFO WARN WARNING ERROR CRITICAL	No
python-version	Virtual environment Python version to use.	No
O	Compile optimized pyc code (see python documentation). Repeat for more optimization.	No
target	The target folder for the Task's virtual environment and source code.	No

config

Use the config subcommand to list your Trains Agent configuration.

Syntax

trains-agent config

daemon

Use the daemon subcommand to run a worker, optionally in a Docker container, listening to a queue.

Syntax

trains-agent daemon [-h] [--foreground] [--queue QUEUES [QUEUES ...]]
                           [--standalone-mode] [--detached] [-O]
                           [--git-user GIT_USER] [--git-pass GIT_PASS]
                           [--log-level {DEBUG,INFO,WARN,WARNING,ERROR,CRITICAL}]
                           [--docker [DOCKER [DOCKER ...]]] [--gpus GPUS]
                           [--cpu-only] [--force-current-version]

Arguments

Argument	Description	Mandatory
cpu-only	If running in Docker mode (see the docker option), disable GPU access for the Docker container.	No
detach	Run in the background. The trains-agent command returns immediately.	No
docker	Run in Docker mode. Execute the Task inside a Docker container. To specify the image name and optional arguments, either: use --docker <image_name> <args> on the command line, or use --docker on the command line, and specify the image name and arguments in the configuration file, see the agent section on the "Trains Agent Reference" page. To ignore the gpus flag inside the Docker container, set the TRAINS_DOCKER_SKIP_GPUS_FLAG environment variable. This also allows you to execute Trains Agent using Docker versions earlier than 19.03. To limit GPU visibility for Docker, set the NVIDIA_VISIBLE_DEVICES environment variable.	No
force-current-version	To use your current version of Trains Agent when running in Docker mode (the docker argument is specified), instead of the latest Trains Agent version which is automatically installed, specify force-current-version. For example, if your current Trains Agent version is 0.13.1, but the latest version of Trains Agent is 0.13.3, use --force-current-version and your Task will execute in the Docker container with Trains Agent version 0.13.1.	No
foreground	Pipe full log to stdout/stderr. Do not use this option if running in background.	No
git-pass	Git password for repository access.	No
git-user	Git username for repository access.	No
gpus	If running in Docker mode (see the docker option), specify the active GPUs for the Docker containers to use. These are the same GPUs set in the NVIDIA_VISIBLE_DEVICES environment variable. For example: --gpus 0 --gpu 0,1,2 --gpus all	No
log-level	SDK log level. The values are: DEBUG INFO WARN WARNING ERROR CRITICAL	No
O	Compile optimized pyc code (see python documentation). Repeat for more optimization.	No
queue	Specify the queues which the worker is listening to. The values can be any combination of: One or more queue Ids. One or more queue names. default indicating the default queue.	No
standalone-mode	Do not use any network connects. This assumes all requirements are pre-installed.	No

execute

Use the execute subcommand to build and execute an experiment without a queue.

Syntax

trains-agent execute [-h] --id TASK_ID [--log-file LOG_FILE]
                            [--disable-monitoring] [--full-monitoring]
                            [--require-queue] [--standalone-mode]
                            [--docker [DOCKER [DOCKER ...]]] [--clone] [-O]
                            [--git-user GIT_USER] [--git-pass GIT_PASS]
                            [--log-level {DEBUG,INFO,WARN,WARNING,ERROR,CRITICAL}]

Arguments

Argument	Description	Mandatory
id	The Id of the Task to build.	Yes
clone	Clone the Task specified by id, and then execute that cloned Task.	No
docker	Run in Docker mode. Execute the Task inside a Docker container. To specify the image name and optional arguments, either: use --docker <image_name> <args> on the command line, or use --docker on the command line, and specify the default image name and arguments in the configuration file, see the agent section on the "Trains Agent Reference" page. To ignore the gpus flag inside the Docker container, set the TRAINS_DOCKER_SKIP_GPUS_FLAG environment variable. This also allows you to execute Trains Agent using Docker versions earlier than 19.03. To limit GPU visibility for Docker, set the NVIDIA_VISIBLE_DEVICES environment variable.	No
disable-monitoring	Disable logging and monitoring, except for stdout.	No
full-monitoring	Create a full log, including the environment setup log, Task log, and monitoring, as well as stdout.	No
git-pass	Git password for repository access.	No
git-user	Git username for repository access.	No
h, help	Get help for this command.	No
log-file	The log file for Task execution output (stdout / stderr) to a text file.	No
log-level	SDK log level. The values are: DEBUG INFO WARN WARNING ERROR CRITICAL	No
O	Compile optimized pyc code (see python documentation). Repeat for more optimization.	No

list

Use the list subcommand to list information about all workers

Syntax

trains-agent list

Use Case Examples

Running worker daemons with queues

The default queue

Run a worker daemon listening to the default queue to fetch Tasks and execute them:
```
trains-agent daemon --queue default
```

User-created queues

Run a worker daemon listening to user-created queued to fetch Tasks and execute them:
```
trains-agent daemon --queue <your_queue>
```

Prioritizing queues

Prioritize queues by specifying more than one queue after the queue option. The order of queues in the command determines the priorities of the queues (first queue is the highest priority).
```
trains-agent daemon --queue important_jobs default
```
The Trains Agent will first try to pull jobs from the important_jobs queue, only then it will fetch a job from the default queue.

Docker mode

Run a worker daemon in a Docker container:

trains-agent daemon --queue default --docker

Use your current Trains Agent version in the Docker container to execute your Task, instead of the latest Trains Agent version which is automatically installed.
```
trains-agent daemon --queue default --docker --force-current-version
```
Run a worker daemon in a Docker container ignoring the gpus flag (also run a worker daemon in Docker version earlier than version 19.03).
Set the environment variable TRAINS_DOCKER_SKIP_GPUS_FLAG = true.
For Kubernetes, specify a host mount on your daemon host. Do not use the host mount inside the Docker container.
Set the environment variable TRAINS_AGENT_K8S_HOST_MOUNT.

For example:
```
TRAINS_AGENT_K8S_HOST_MOUNT=/mnt/host/data:/root/.trains
```

Specifying GPUs

Run a worker daemon with more than one GPU on the same machine:

trains-agent daemon --gpus 0 --queue default &
trains-agent daemon --gpus 1 --queue default &

Run a worker daemon listening with more than one GPU and a dedicated queue:

trains-agent daemon --gpus 0,1 --queue dual_gpu &
trains-agent daemon --gpus 2,3 --queue dual_gpu &

Debugging

Running Trains Agent in foreground mode sending all output to the console:
```
trains-agent daemon --queue default --foreground
```

Executing Tasks without queues

Execute a Task without queue

Execute a Task in a worker without a queue:
```
trains-agent execute --id <task-id>
```

Clone a Task and execute the cloned Task

Clone the specified Task and execute the cloned Task in worker without a queue:
```
trains-agent execute --id <task-id> --clone
```

Docker mode

Execute a Task in worker using a Docker container without a queue:
```
trains-agent execute --id <task-id> --docker
```