Comparing Experiments

The ClearML Web UI provides a deep experiment comparison, allowing you to locate, visualize, and analyze differences including:

  • Source code - Repository, branch, commit ID, script, and working directory.

  • Artifacts - Input model, output model, and model design.

  • Hyperparameters - Side by side values comparison, and hyperparameter parallel coordinate comparison.

  • Scalars - Compare specific values, and plots of scalar series.

  • Plots - Compare plots of any series data.

  • Debug samples - Compare debug samples by each iteration. Examine samples with a viewer (for images and video), and a player (for audio).

The ClearML experiment comparison provides comparison features making it easy to compare experiments.

Selecting experiments to compare

To select experiments to compare:

  1. Select the experiments using multi-select, select the checkboxes individually, select the top checkbox for all experiments. After selecting the second checkbox, the bottom bar appears.

  2. In the bottom bar, click COMPARE. The comparison page appears, showing a column for each experiment and differences with a highlighted background color. The experiment on the left is the base experiment. Other experiments compare to the base experiment.

Source code and artifacts differences

Source code and artifact differences appear in the DETAILS tab. These are deep comparisons and include:

  • Source code

    • The repository, branch, commit ID, script file name, and working directory.

    • Installed packages and versions, sorted by package name.

    • Uncommitted changes, sorted by file name.

  • Artifacts

    • Input model and model design.

    • Output model and model design.

    • Other artifacts, if any.

To locate the source differences:

  • Click the DETAILS tab > Expand highlighted sections, or in the header, click (Previous diff) or (Next diff).

For example, in the image below, expanding ARTIFACTS > Output Model > Model, and EXECUTION > Uncommitted Changes shows the model ClearML model ID and name are different, and the location of the uncommitted change differences.

image

Hyperparameters

Compare hyperparameters as values, or compare by metric (hyperparameter parallel coordinate comparison).

Side by side values comparison

The comparison shows hyperparameter value differences highlighted line-by-line.

To view a side by side values comparison:

  1. Click the HYPER PARAMETERS tab.

  2. In the list (upper right), choose Values.

  3. You can move the Hide Identical Fields slider to on, and show only differences.

  4. Locate differences by either:

    • Clicking (Previous diff) or (Next diff).

    • Scrolling to see highlighted hyperparameters.

For example, by expanding General shows the batch_size and epochs different between the experiments.

image

Compare by metric

Compare a metric to any combination of hyperparameters using a parallel coordinates plot.

To compare by metric:

  1. Click the HYPER PARAMETERS tab.

  2. In the list (upper right), choose Parallel Coordinates.

  3. In Performance metrics, expand a metric or monitored resource, and then click a variant.

  4. Select the metric values to use. Choose one of the following:

    • LAST - The final value, or the most recent value, for in-progress experiments.

    • MIN - Minimal value.

    • MAX - Maximal value.

  5. In Parameters, and then select all hyperparameter checkboxes to compare.

  6. To view one experiment on the plot, hover over the experiment name in the legend.

For example, plot the metric/variant epoch_accuracy/validation: epoch_accuracy plotted against the hyperparameters batch_size and epochs.

image

Hover over one of the experiment names in the legend, and the plot shows only that data.

image

Scalars

Visualize the comparison of scalars which includes metrics and monitored resources in the SCALARS tab.

Compare specific values

To compare specific values:

  1. Click the SCALARS tab.

  2. In the list (upper right), choose either:

    • Last values (the final or most recent value)

    • Min Values (the minimal values)

    • Max Values (the maximal values)

  3. Sort by variant.

image

Compare scalar series

Compare scalar series in plots and analyze differences using ClearML Web UI plot tools.

To compare scalar series:

  1. Click the SCALARS tab.

  2. In the list (upper right), Graph.

To improve scalar series analysis:

  • Group scalars by metric (all variants for a metric on the same plot) or metric-variant combination (individual metric-variant plots). - In Group by, select Metric or Metric+Variant.

  • Horizontal axis options - Iterations, relative time since the experiment began, wall (clock time). Click , and then click an option.

  • Smooth a curve - Click , and move the Smoothing slider or type a smoothing number from 0 to 0.999.

  • Use plot controls, which appear when you hover over the top of a plot.

  • Hide / show scalar plots - Click Hide all and .

  • Filter scalars by full or partial scalar name.

This image shows scalars grouped by metric.

image

This image shows scalars grouped by metric and variant.

image

Plots

Visualize the comparison of any data ClearML automatically captures, and which you explicitly report in your experiments, in the PLOTS tab.

To compare plots:

  1. Click the PLOTS tab.

  2. To improve your comparison, use either of the following:

    • To locate scalars, you can click HIDE ALL , and then (show) for those you want to see. You can also filter scalars by full or partial scalar name.

    • Use any of the plot controls which appear when you hover over the top of a plot, including downloading the image, downloading the data as JSON, zoom, pan, and logarithmic scale.

Debug samples

Compare debug samples at any iteration to verify your experiment is running as expected. The most recent iteration appears first. Use the viewer / player to inspect images, audio, video samples and do any of the following:

  • Move to the same sample in a different iteration (move the iteration slider).

  • Show the next or previous iteration’s sample.

  • Download the file ().

  • Zoom.

  • View the sample’s iteration number, width, height, and coordinates.

To compare debug samples:

  1. Click the DEBUG SAMPLES tab. The most recent iteration appears at the top.

  2. Locate debug samples by doing the following:

    • Filter by metric. In the Metric list, choose a metric.

    • Show other iterations. Click image (Older images), image (New images), or image (Newest images).

    image

  3. To open a debug sample (image, audio, or video) in the viewer or player, click the thumbnail.

    image

  4. To move to the same sample in another iteration, click < (previous), > (next), or move the slider.

To view a debug sample in the viewer / player:

  1. Click the debug sample click the thumbnail.

  2. Do any of the following:

    • Move to the same sample in another iteration - Move the slider, or click < (previous) or > (next).

    • Download the file - Click .

    • Zoom

    • For images, locate a position on the sample - Hover over the sample and the X, Y coordinates appear in the legend below the sample.

Comparison features

To assist you in your analysis, the comparison page supports:

Adding experiments to the comparison

Add an experiment to the comparison - Click Add Experiment and start typing an experiment name. An experiment search and select dialog appears showing matching experiments to choose from. To add an experiment, click +. To Remove an experiment, click .

image

Find the next or previous difference

  • Find the previous difference (), or the next difference ().

Hiding identical fields

  • Move the Hide Identical Fields slider to on.

Search all text

Search all text in the comparison.

Choosing a different base experiment

Set a new base experiment. Show differences in other experiments to the new base.

image

Dynamic ordering of the compared experiments

  • Drag the experiment to a new position

image

Remove an experiment from the comparison

Remove an experiment from the comparison.

image