Tracking Experiments and Visualizing Results
Trains is now ClearML
This documentation applies to the legacy Trains versions. For the latest documentation, see ClearML.
While an experiment is running, and any time after it finishes, you can track it and visualize the results in the Trains Web-App (UI).
The experiment info panel
Tracking information and results appear in the experiment info panel. The info panel slides open when you click an experiment. When the info panel is open, you can no longer see all experiment table columns. The only column that appears (to the right of the info panel) is the column currently used to sort the table. That column becomes drop list of all sortable columns. The filter () becomes a list of all filterable columns.
Experiment details are organized in the following info panel tabs:
- Tracking
- EXECUTION tab - The experiment source code, the base Docker image for Trains Agent, output destination for artifact storage, and log level.
- CONFIGURATION tab - Hyperparameters, user properties, and configuration objects.
- ARTIFACTS tab - Artifacts
- INFO tab - General experiment information.
- RESULTS tab - Experiment results, including the log, scalars and other plots, and debug samples.
The info panel also shows tags, which are user defined, color-coded descriptive labels you can apply to experiments (and to models). These tags also appear in the experiment table when the info panel is closed.
Tracking
Execution
Source code
Source code information includes:
- The experiment's repository, commit ID, script path, and working directory.
- Uncommitted changes.
- Installed Python packages and their versions.
Base Docker image
The base Docker image is a pre-configured Docker that Trains Agent will use to remotely execute this experiment (see the Building Docker containers).
Artifact output destination
An output destination you specify for storing model checkpoints (snapshots) and artifacts (see also, default_output_uri in the configuration file, and output_uri in Task.init
parameter)
Log level
The logging level for the experiment (see the standard Python logging levels.
Configuration
All parameters and configuration objects appear in the CONFIGURATION tab.
Hyperparameters
In older versions of Trains Server, the CONFIGURATION tab was named HYPER PARAMETERS, and it contained all parameters. The renamed tab contains a HYPER PARAMETER section, and subsections for hyperparameter groups.
Hyperparameters are grouped by their type. They are in the HYPER PARAMETERS section of the CONFIGURATION tab, and are grouped as follows:
-
Command line arguments. In the Args section. Automatically logged
argparse
arguments, and all older experiments parameters, except TensorFlow DEFINEs. Hover over a parameter, and the type, description, and default value appear, if they were provided.View a screenshot
-
TensorFlow defintions. In the TF_DEFINE section. Automatic TensorFlow logging.
View a screenshot
-
Parameter dictionaries. In the General section. See connecting a
dict
object when calling the Task.connect method.View a screenshot
-
Environment variables. In the Environment section. Logged if you set the
TRAINS_LOG_ENVIRONMENT
environment variable, see this FAQ).View a screenshot
-
Custom named parameter groups. See the
name
parameter when calling the Task.connect method.View a screenshot
Once an experiment is run and stored in Trains Server, any of these hyperparameters may have been set by modifying the experiment.
User properties
User properties allow you to store any descriptive information in key-value pair format. They are editable in any experiment, except experiments whose status is Published (read-only).
View a screenshot
Configuration objects
Trains tracks experiment (Task) model configuration objects, which appear in Configuration Objects > General. This includes automatic tracking, and those connected to a Task in code (see Task.connect_configuration). Trains supports providing a name for a Task model configuration (see the name parameter in Task.connect_configuration
.
In older versions of Trains Server, the Task model configuration appeared in the ARTIFACTS tab, MODEL CONFIGURATION section. Task model configurations now appear in the Configuration Objects section, in the CONFIGURATION tab.
View a screenshot
Artifacts
Artifacts tracked with an experiment appear in the ARTIFACTS tab and include models and other artifacts.
Models
The input and output model appear in the ARTIFACTS tab. It shows the model names, their model designs, and the name of the experiment that created the model (stored it in Trains Server). In Trains, models are associated with the experiment, but the model details, including the model design, class enumeration, and general information about the model are in the MODELS tab. The model name is a hyperlink to those details.
View a screenshot
Data audit
Artifacts which are uploaded and dynamically tracked by Trains appear in the DATA AUDIT section. They include the file path, file size, hash, and metadata stored with the artifact.
View a screenshot
Other
Other artifacts, those which are uploaded but not dynamically tracked after the upload, appear in the OTHER section. They include the file path, file size, and hash.
View a screenshot
General information
General experiment is in the INFO tab. This includes information describing the stored experiment, including creation, start, and last update date and time, the user creating the experiment, the project name, experiment state (status), and whether the experiment is archived.
Experiment results
All experiment results appear in the RESULTS tab, including:
- The experiment log.
- Scalar plots and other plots which are automatically logged by Trains, as well as plots explicitly reported in code, appear in SCALARS > PLOTS.
- Image, audio, video, and text file debug samples.
Viewing modes
The Trains Web (UI) supports two viewing modes for results: full screen (full screen width showing results only) and info panel (adjustable width sliding panel showing all experiment details).
To switch between viewing modes:
-
From the info panel to full screen - Click
(View in experiment table), or in the menu (
) > Results.
View a screenshot
-
From full screen to the info panel - Click
(View in experiment table).
View a screenshot
Log
The complete experiment log containing everything printed to stdout and strerr is in the LOG tab. The full log is downloadable. When viewing the log, you can click Jump to end.
Scalars and other plots
All scalars that Trains automatically logs, as well as those explicitly reported in code, appears in RESULTS > SCALARS tab. All other plots appears in the PLOTS sub tab.
In results full screen mode, the Trains Web (UI) provides the following plot features:
- Show / hide plots - Click HIDE ALL , and then (show) those you want to see.
- Horizontal axis modes (scalars, only) - Click > Select ITERATIONS, RELATIVE (time since experiment began), or WALL (local clock time).
- Curve smoothing (scalars, only) - Click > Slide the slider or type a smoothing factor between 0 and 0.999.
Each plot supports plot controls allowing you better analyze the results. The table below lists the plot controls which may be available for any plot. Hover over a plot, and the controls appear.
Icon | Description |
---|---|
Download plots as PNG files. | |
Pan around plot. Click |
|
To examine an area, draw a dotted box around it. Click |
|
To examine an area, draw a dotted lasso around it. Click |
|
Zoom into a section of a plot. Zoom in - Click |
|
Zoom in. | |
Zoom out. | |
Reset to autoscale after zooming ( |
|
Reset axes after a zoom. | |
Show / hide spike lines. | |
Show the closest data point on hover, including horizontal and vertical axes values. Click |
|
Compare data on hover. Click |
|
Switch to logarithmic view. | |
Hide / show the legend. | |
To get metric data for further analysis, download plot data to JSON file. |
Debug samples
Debug samples automatically logged by Trains and those explicitly reported in code appear in RESULTS > DEBUG SAMPLES. Debug samples appear by metric and iteration within each metric. You can view debug samples for other iterations by clicking (Older images),
(New images), or
(Newest images).
Debug samples include:
- Images - See the height and width in pixels, zoom, and download the image in the image viewer.
- Audio - Play the audio and download it in the audio player.
- Video - Play the video and download it in the video player.
- Text files - Copy the text file URL and open the text file in a new browser tab.
For images, audio, and video, click the thumbnail to open the viewer / player.
Tagging experiments
Tags are user-defined, color-coded labels that you can add to experiments (and models) allowing you to easily identify and group of experiments. A tag shows any text you choose, for any purpose. For example, add tags for the type of remote machine experiments execute on, label versions of experiments, or apply team names to organize experimentation.
- Adding tags and changing tag colors:
- Click the experiment > Hover over the tag area > +ADD TAG or
(menu)
- Do one of the following:
- Add a new tag - Type the new tag name > (Create New).
- Add an existing tag - Click a tag.
- Change a tag's colors - Click Tag Colors > Click the tag icon > Background or Foreground > Pick a color > OK > CLOSE.
- Click the experiment > Hover over the tag area > +ADD TAG or
- Remove a tag - Hover over the tag > X.
Locating the experiment (Task) ID
- In the info panel, top area, click ID. The Task ID appears.