Trains Agent Reference
Trains is now ClearML
This documentation applies to the legacy Trains versions. For the latest documentation, see ClearML.
This reference page provides details information about Trains Agent commands, including:
- build - Create a worker environment, without executing an experiment.
- config - List your Trains Agent configuration.
- daemon - Run a worker daemon listening to a queue for Task (experiments) to execute.
- execute - Execute an experiment, locally without a queue.
- list - List the current workers.
build
Use the build
subcommand to create a worker environment, but do not execute the experiment.
Syntax
trains-agent build [-h] --id TASK_ID [--target TARGET]
[--install-globally]
[--docker [DOCKER [DOCKER ...]]]
[--python-version PYTHON_VERSION]
[--entry-point {reuse_task,clone_task}] [-O]
[--git-user GIT_USER] [--git-pass GIT_PASS]
[--log-level {DEBUG,INFO,WARN,WARNING,ERROR,CRITICAL}]
[--gpus GPUS] [--cpu-only]
Arguments
id
(mandatory)
- Build a worker environment for this Task ID.
cpu-only
- Disable GPU access for the Docker container.
docker
-
Docker mode. A Docker container that a worker will execute at launch.
To specify the image name and optional arguments, either:
- use
--docker <image_name> <args>
on the command line, or - use
--docker
on the command line, and specify the image name and arguments in the configuration file, see the agent section on the "Trains Agent Reference" page.
Environment variable settings for Dockers:
TRAINS_DOCKER_SKIP_GPUS_FLAG
- Ignore thegpus
flag inside the Docker container. This also allows you to execute Trains Agent using Docker versions earlier than 19.03.NVIDIA_VISIBLE_DEVICES
- Limit GPU visibility for the Docker container.TRAINS_AGENT_GIT_USER
andTRAINS_AGENT_GIT_PASS
- Pass these credentials to the Docker container at execution.
To limit GPU visibility for Docker, set the
NVIDIA_VISIBLE_DEVICES
environment variable. - use
entry-point
-
Used in conjunction with
--docker
, indicates how to run the Task specified bytask-id
on Docker startup.The
entry-point
options are:reuse
- Overwrite the existing Task data.clone_task
- Clone the Task, and execute the cloned Task.
git-pass
- Git password for repository access.
git-user
- Git username for repository access.
gpus
-
Specify the active GPUs for the Docker containers to use. These are the same GPUs set in the
NVIDIA_VISIBLE_DEVICES
environment variable.For example:
--gpus 0
--gpu 0,1,2
--gpus all
h
, help
- Get help for this command.
install-globally
- Install the required Python packages before creating the virtual environment. Use
agent.package_manager.system_site_packages
to control the installation of the system packages. When--docker
is used,install-globally
is always true.
log-level
-
SDK log level. The values are:
DEBUG
INFO
WARN
WARNING
ERROR
CRITICAL
python-version
- Virtual environment Python version to use.
O
- Compile optimized pyc code (see python documentation). Repeat for more optimization.
target
- The target folder for the virtual environment and source code that will be used at launch.
config
Use the config
subcommand to list your Trains Agent configuration.
Syntax
trains-agent config
daemon
Use the daemon
subcommand to run a worker, optionally in a Docker container, listening to a queue.
Syntax
trains-agent daemon [-h] [--foreground] [--queue QUEUES [QUEUES ...]]
[--order-fairness] [--standalone-mode]
[--services-mode] [--create-queue] [--detached]
[--stop] [-O] [--git-user GIT_USER]
[--git-pass GIT_PASS]
[--log-level {DEBUG,INFO,WARN,WARNING,ERROR,CRITICAL}]
[--gpus GPUS] [--cpu-only]
[--docker [DOCKER [DOCKER ...]]]
[--force-current-version]
Arguments
cpu-only
- If running in Docker mode (see the
docker
option), disable GPU access for the Docker container.
create-queue
- If the queue name provided does not exist in the server, create and use it.
detached
- Run in the background. The
trains-agent
command returns immediately.
docker
-
Run in Docker mode. Execute the Task inside a Docker container.
To specify the image name and optional arguments, either:
- use
--docker <image_name> <args>
on the command line, or - use
--docker
on the command line, and specify the image name and arguments in the configuration file, see the agent section on the "Trains Agent Reference" page.
Environment variable settings for Dockers:
TRAINS_DOCKER_SKIP_GPUS_FLAG
- Ignore thegpus
flag inside the Docker container. This also allows you to execute Trains Agent using Docker versions earlier than 19.03.NVIDIA_VISIBLE_DEVICES
- Limit GPU visibility for the Docker container.TRAINS_AGENT_GIT_USER
andTRAINS_AGENT_GIT_PASS
- Pass these credentials to the Docker container at execution.
- use
force-current-version
-
To use your current version of Trains Agent when running in Docker mode (the
docker
argument is specified), instead of the latest Trains Agent version which is automatically installed, specifyforce-current-version
.For example, if your current Trains Agent version is
0.13.1
, but the latest version of Trains Agent is0.13.3
, use--force-current-version
and your Task will execute in the Docker container with Trains Agent version0.13.1
.
foreground
- Pipe full log to stdout/stderr. Do not use this option if running in background.
git-pass
- Git password for repository access.
git-user
- Git username for repository access.
gpus
-
If running in Docker mode (see the
docker
option), specify the active GPUs for the Docker containers to use.These are the same GPUs set in the
NVIDIA_VISIBLE_DEVICES
environment variable.For example:
--gpus 0
--gpu 0,1,2
--gpus all
log-level
-
SDK log level. The values are:
DEBUG
INFO
WARN
WARNING
ERROR
CRITICAL
O
- Compile optimized pyc code (see python documentation). Repeat for more optimization.
order-fairness
- Round-robin queue polling.
queue
-
Specify the queues which the worker is listening to. The values can be any combination of:
- One or more queue Ids.
- One or more queue names.
default
indicating the default queue.
- Spin multiple, simultaneous Tasks, each in its own Docker container, on the same machine. Each Task will be registered as a new node in the system, providing tracking and transparency capabilities. Start up and shutdown of each Docker is verified. Use in CPU mode (
--cpu-only
), only.
standalone-mode
- Do not use any network connects. This assumes all requirements are pre-installed.
stop
- Terminate a running Trains Agent, if other arguments are the same. If no additional arguments are provided, agents are terminated in lexicographical order.
execute
Use the execute
subcommand to build and execute an experiment without a queue.
Syntax
trains-agent execute [-h] --id TASK_ID [--log-file LOG_FILE]
[--disable-monitoring] [--full-monitoring]
[--require-queue] [--standalone-mode]
[--docker [DOCKER [DOCKER ...]]] [--clone] [-O]
[--git-user GIT_USER] [--git-pass GIT_PASS]
[--log-level {DEBUG,INFO,WARN,WARNING,ERROR,CRITICAL}]
Arguments
id
(mandatory)
- The ID of the Task to build.
clone
- Clone the Task specified by
id
, and then execute that cloned Task.
docker
-
Run in Docker mode. Execute the Task inside a Docker container.
To specify the image name and optional arguments, either:
- use
--docker <image_name> <args>
on the command line, or - use
--docker
on the command line, and specify the default image name and arguments in the configuration file, see the agent section on the "Trains Agent Reference" page.
Environment variable settings for Dockers containers:
TRAINS_DOCKER_SKIP_GPUS_FLAG
- Ignore thegpus
flag inside the Docker container. This also allows you to execute Trains Agent using Docker versions earlier than 19.03.NVIDIA_VISIBLE_DEVICES
- Limit GPU visibility for the Docker container.TRAINS_AGENT_GIT_USER
andTRAINS_AGENT_GIT_PASS
- Pass these credentials to the Docker container at execution.
- use
disable-monitoring
- Disable logging and monitoring, except for stdout.
full-monitoring
- Create a full log, including the environment setup log, Task log, and monitoring, as well as stdout.
git-pass
- Git password for repository access.
git-user
- Git username for repository access.
h
, help
- Get help for this command.
log-file
- The log file for Task execution output (stdout / stderr) to a text file.
log-level
-
SDK log level. The values are:
DEBUG
INFO
WARN
WARNING
ERROR
CRITICAL
O
- Compile optimized pyc code (see python documentation). Repeat for more optimization.
list
Use the list
subcommand to list information about all workers
Syntax
trains-agent list