Trains Python Package Reference

Trains is now ClearML

This documentation applies to the legacy Trains versions. For the latest documentation, see ClearML.

Overview

This reference section provides detailed information about the Trains Python Package classes and their usage. This section also includes Trains Python Package Extras which are additional Trains components that require installation and configuration separate from the Trains Python Package. These classes and extras allow you to implement the many features of Trains for enhanced experimentation and automation. This is in addition to our automagical capturing of experiment inputs and outputs.

Task Module

The Task Module reference page provides detailed information for task.py in the trains repository.

The Task class contains methods to manage all the parts of experiments and integrate them with Trains for tracking, storage, coding, analysis, comparison, and experiment management, including:

  • Task.init - Initialize a Task (experiment).
  • connect - Connect the parts of an experiment to the experiment in Trains, including hyperparameter dictionaries, input and output models, configuration dictionaries, and class enumeration labels.
  • get_logger - Create a logger object which you can use for explicit reporting.
  • get_task - Get a reference to any experiment in Trains by Task Id, task name and project name, or task name only.
  • register_artifact - Register an artifact, and Trains logs changes to it.
  • get_registered_artifacts - After you register an artifact, retrieve a reference to it, manipulate the artifact in your code, and the changes are updated in Trains.
  • upload_artifact - Upload an artifact, but Trains does not logs changes to it.
  • get_model_config_dict - Get the configuration as a dictionary and then easily work with it in your code.
  • set_model_config - Set the model configuration before creating an output model so that the output model inherits the configuration properties.
  • set_model_label_enumeration - Similar to set_model_config(), but for class label enumeration.
  • clone - Create an editable copy of an experiment. For example, clone an experiment for hyperparameter tuning or multiple times for automation.
  • enqueue - Add an experiment to a queue. Similar to dequeue() above, enqueue() allows you to develop scripts which can automate your experimentation.
  • dequeue - Remove an enqueued experiment from a queue.
  • reset - Make an experiment editable and allow remote execution of the experiment. This deletes output of the previous.
  • set_initial_iteration - To continue a previous experiment run, set the initial iteration (set to a non-zero).
  • get_initial_iteration - Get the value of the initial iteration.

Logger Module

The Logger Module reference page provides detailed information for logger.py in the trains repository.

Use the Logger class and methods to extend Trains automagical capturing of stdout and stderr with Trains' own explicit reporting for scalar metrics, any plot data in a variety of chart types, text messages, errors, warnings, debug messages, uploading tables, and uploading debug samples (images, audio, and video), as well as other methods.

  • Logger.current_logger
  • Logger.report_text
  • Logger.report_scalar
  • Logger.report_line_plot
  • Logger.report_scatter2d
  • Logger.report_scatter3d
  • Logger.report_histogram
  • Logger.report_table
  • Logger.report_confusion_matrix
  • Logger.set_upload_destination
  • Logger.report_image
  • Logger.report_media

Model Module

The Model Module reference page provides detailed information for model.py in the trains repository.

Use the Model class's InputModel and OutputModel classes and methods to augment control of input models and output using methods such as:

  • InputModel.import_model - Import a pre-trained model for your experiment.
  • InputModel.connect - Connect the current model the Task (experiment).
  • InputModel.config_dict - Get the input model's configuration as a dictionary.
  • InputModel.get_weights - Download the base model.
  • OutputModel.connect - Connect the output model to the Task (experiment).
  • OutputModel.set_upload_destination - Specify a destination for uploading debug samples, including images, audio, and video.

Automation Module

The Automation Module reference page provides detailed information for the following:

Storage Module

The Storage Module reference page provides detailed information for manager.py in the trains repository.

To list, download, and upload files to / from storage, as well as manage cache, use the StorageManager and StoreHelper class methods.

Trains Python Package Extras

Trains Python Package Extras are components of the Trains Python Package which provide additional features, but are not installed when you install Trains (meaning, they are not installed when you execute, pip install trains). They are referred to as extras, because they are defined in the Trains Python Package setup.py extras_require dictionary. Python package components in the extras_require dictionary require separate installation.

The Trains Python Package Extras include extras for experiment storage of model checkpoints (snapshots) and other artifacts. The supported Cloud storage types include:

  • AWS S3
  • Google Cloud Storage
  • Azure Storage

These extras for storage also require that you add your storage bucket names, credentials, and related information to your Trains configuration file.

To install and configure any of these, follow the instructions on the Trains Python Package Extras page.

After installing and configuring, you can use the Cloud storage in your Python experiment scripts by calling the Task.init method, and specifying the output_uri parameter.