Tracking Leaderboards

In this tutorial, you learn how to set up a tracking leaderboard. A tracking leaderboard allows you to easily monitor your experiments using a customized experiments table with auto refresh for continual updates.

You can customize your tracking leaderboard with information including experiment status, elapsed time since the experiment started and last updated, most recent iteration, metrics, and hyperparameters.

Also, bookmark the leaderboard in your browser for reuse and share the leaderboard URL with other users.

For this tutorial, use one of the following as your project:

image

Step 1. Select a project

The leaderboard will track experiments in one or more projects. For this, we begin by opening the experiments table for a specific project or all projects.

To select a project, do the following:

  • In the Trains Web-App (UI), open a project's experiments table. Do either of the following:

    • On the Home page, click a project card or VIEW ALL.
    • On the Projects page, click project card or the All projects card.

Step 2. Filter the experiments

The experiments table allows you to filter experiments by experiment name, type, and status.

To filter experiments, do the following:

Apply any combination of the following filters:

  • Name - Use the search to find experiments containing any search term.
  • Type - Click the TYPE column's filter and then on the list select Training and / or Testing.
  • Status - Click the STATUS column's filter and then on the list select any of the following:

    • Draft - The experiment is not running.
    • Pending - The experiment is in a queue waiting to be run by a worker.
    • Running - The experiment is currently running by a worker.
    • Completed - The experiment ran and terminated successfully.
    • Failed - The experiment ran and terminated with an error.
    • Aborted - The experiment ran and was manually or programmatically terminated.
    • Published - The experiment is not running, it is preserved as read-only.

Step 3. Hide the defaults column

Customize the columns on your tracking leaderboard by hiding any of the default columns shown below.

To select the defaults, do the following:

  1. Click settings.
  2. Select / clear any combination of the default columns of the following:

    • TYPE - The type of experiment, training or testing.
    • NAME - The name of the experiment.
    • STATUS - The status of the experiment.
    • PROJECT - The project name.
    • USER - The user creating or running the experiment.
    • STARTED - The elapsed time since the experiment started running.
    • UPDATED - The elapsed time since the experiment update.
    • ITERATION - The last iteration of the experiment. For experiments with a status of Running, this is the most recent iteration. For Completed, Aborted, and Failed experiments, this is the final iteration.

Step 4. Show metrics or hyperparameters

The leaderboard can contain any combination of metrics and hyperparameters. For each metric, you can choose the last (most recent), minimum, and / or maximum values.

To select metrics or hyperparameters, do the following:

  1. Click settings.
  2. For metrics, click + METRIC, expand a metric category, and then select the metric checkboxes of those to show in the leaderboard.
  3. For hyperparameters, click + HYPER PARAMETERS, and then select the hyperparameter checkboxes of those to show in the leaderboard.

Step 5. Enable auto refresh

Auto refresh allows you monitor the progress of your experiments. It is enabled by default. If auto refresh is disabled, enable it.

To enable auto refresh, do the following:

  • Hover over refresh and then check the Auto Refresh checkbox.

Step 6. Save the tracking leaderboard

The URL for Trains Web-App (UI) now contains parameters and values for the customized leaderboard. Bookmark it and you can return to it for monitoring.

You can also share the leaderboard with another user by copying the URL to the clipboard and sharing the URL.