Tuning Experiments

In this tutorial, you learn how to tune an experiment using the Trains Web-App. We use the Python experiment script tensorflow_mnist_with_summaries.py which is in the trains repository to create an original experiment, tune its hyperparameters, and then compare the original to the tuned experiment.

Prerequisites

Step 1. Run the initial experiment

In your local trains repository, examples directory, run the experiment script:

python tensorflow_mnist_with_summaries.py

How to view the experiment in the Trains Web-App

For more information about viewing the experiment, see Experiment Details in the User Interface section.

Step 2. Clone the initial experiment

Create an editable copy of the experiment so that we can tune it. Do the following:

  1. In the Trains Web-App, on the Projects page, click the examples project card.
  2. In the experiments table, right click the experiment Tensorflow mnist with summaries example.

    If more than one experiment with the same name is in the experiments table

    Another user may have run this example previously. If another experiment with the same name is in the experiments, sort the table by the UPDATED column.

  3. In the context menu, click Clone.
  4. In the dialog window, accept the default values for the cloned experiment's project, name, and description. "Clone Of" is prepended to the name.
  5. Click CLONE. The cloned experiment is automatically selected and its details panel slides open with the EXECUTION tab open.

Step 3. Tune the cloned experiment

Change two hyperparameter values. Do the following:

  1. In the experiment details panel (which opened in the previous step), click the HYPER PARAMETERS tab.
  2. Hover over the HYPER PARAMETERS area and then click EDIT.
  3. Locate the hyperparameter dropout and change its value from 0.9 to 0.7.
  4. Locate the hyperparameter learning_rate and change its value from 0.001 to 0.05.
  5. Click SAVE.

Step 4. Run a worker daemon listening to a queue

To run the tuned experiment, we need a worker daemon listening to the default queue, because we will enqueue the experiment to the default queue. For more information about workers, worker daemons, and queues, see the "Concepts and Architecture" page, Workers and queues and Trains Agent.

Run the worker daemon on your local development machine by doing the following:

  1. Open a terminal session.
  2. Run the following trains-agent command which runs a worker daemon listening to the default queue.
    trains-agent daemon --queue default
    The response to this command is information about your configuration, the worker, and the queue. For example:
    Current configuration (trains_agent v0.13.0, location: /home/<username>/trains.conf):
    ----------------------
    agent.worker_id =
    agent.worker_name = LAPTOP-PPTKKPGK
    agent.python_binary =
    agent.package_manager.type = pip
    .
    .
    .
    sdk.development.worker.report_period_sec = 2
    sdk.development.worker.ping_period_sec = 30
    sdk.development.worker.log_stdout = true
    
    Worker "LAPTOP-PPTKKPGK:0" - Listening to queues:
    +----------------------------------+---------+-------+
    | id                               | name    | tags  |
    +----------------------------------+---------+-------+
    | 2a03daf5ff9a4255b9915fbd5306f924 | default |       |
    +----------------------------------+---------+-------+
    
    Running TRAINS-AGENT daemon in background mode, writing stdout/stderr to /home/<username>/.trains_agent_daemon_outym6lqxrz.txt
    The worker is now listening for enqueued experiments.

Step 5. Enqueue the tuned experiment

Enqueue the tuned experiment. Do the following:

  1. In the Trains Web-App, experiments table, right click the experiment Clone Of Tensorflow mnist with summaries example.
  2. In the context menu, click Enqueue.
  3. If the queue is not Default, in the queue list, select Default.
  4. Click ENQUEUE. The experiment's status becomes Pending. When the worker fetches the experiment from the queue, the status becomes Running, and you can view its progress in the details panel.

Step 6. Compare the experiments

When the tuned experiment finished running, and the status becomes Completed, we can compare the original and the tuned experiments.

To compare the experiments, do the following:

  1. In the Trains Web-App, on the Projects page, click the examples project.

    The Trains demo Web-App

    If you are using the demo Trains Server, the Projects page is https://demoapp.trains.allegro.ai/projects. If you deployed a self-hosted Trains server, open it in your browser.

  2. In the experiments table, select the checkboxes for our two experiments, Tensorflow mnist with summaries example and Clone Of Tensorflow mnist with summaries example.
  3. On the Compare / Archive footer, click COMPARE. The experiment comparison window appears. The experiment comparison window is organized in the following tabs:
    • DETAILS - The ARTIFACTS section, including input and output models with their network designs, and other artifacts; the EXECUTION section execution, including source code control, installed Python packages and versions, uncommitted changes, and the Docker image name which, in this case, is empty.
    • HYPER PARAMETERS - The hyperparameters and their values.
    • SCALARS - Scalar metrics with the option to view them as charts or values.
    • PLOTS - Plots of any data with the option to view them as charts or values.
    • DEBUG SAMPLES - Media including images, audio, and video uploaded by your experiment shown as thumbnails.
    • All differences between the compared experiments appear with a different background color to highlight them.
  4. Examine the differences in our two experiments by doing the following:
    1. Click the DETAILS tab.
    2. Expand the EXECUTION section.
    3. Expand the Parameters section.
    4. The hyperparameters dropout and learning_rate are shown with a different background color. The values are different.
    5. Click the SCALARS tab.
    6. To the right of Add Experiment, click Values. The scalar values appear as values showing the difference highlighted by background color.
    7. To the right of Add Experiment, click Chart. The scalar charts appear showing the difference.

Next Steps