In this tutorial, you learn how to tune an experiment. We tune the experiment which the tensorflow_mnist.py example script creates.
- Clone our trains repository.
- Install the requirements for the TensorFlow examples.
- Trains Agent is installed and configured.
Step 1. Run the experiment
examples/frameworks/tensorflow directory, run the experiment script:
Step 2. Clone the experiment
Clone the experiment to create an editable copy that we can tune it. Do the following:
- In the Trains Web-App (UI), on the Projects page, click the
- In the experiments table, right click the experiment
Tensorflow v2 mnist with summaries.
- In the context menu, click Clone > CLONE. The newly cloned experiment appears and its info panel slides open.
Step 3. Tune the cloned experiment
To demonstrate tuning, change two hyperparameter values. Do the following:
- In the experiment info panel, click the CONFIGURATIONS tab.
- In PARAMETERS > General, hover and then click EDIT.
- Change the value of dropout from
- Change the value of learning_rate from
- Click SAVE
Step 4. Run a worker daemon listening to a queue
To execute the cloned experiment which we tuned, we need a running worker daemon listening to the
Run the worker daemon on your local development machine by doing the following:
Open a terminal session.
Run the following
trains-agentcommand which runs a worker daemon listening to the
trains-agent daemon --queue default
The response to this command is information about your configuration, the worker, and the queue. For example:
Current configuration (trains_agent v0.13.0, location: /home/<username>/trains.conf): ---------------------- agent.worker_id = agent.worker_name = LAPTOP-PPTKKPGK agent.python_binary = agent.package_manager.type = pip . . . sdk.development.worker.report_period_sec = 2 sdk.development.worker.ping_period_sec = 30 sdk.development.worker.log_stdout = true Worker "LAPTOP-PPTKKPGK:0" - Listening to queues: + ---------------------------------+---------+-------+ | id | name | tags | + ---------------------------------+---------+-------+ | 2a03daf5ff9a4255b9915fbd5306f924 | default | | + ---------------------------------+---------+-------+ Running TRAINS-AGENT daemon in background mode, writing stdout/stderr to /home/<username>/.trains_agent_daemon_outym6lqxrz.txt
Step 5. Enqueue the tuned experiment
Enqueue the tuned experiment. Do the following:
- In the Trains Web-App (UI), experiments table, right click the experiment
Clone Of Tensorflow v2 mnist with summaries.
- In the context menu, click Enqueue.
- If the queue is not Default, in the queue list, select Default.
- Click ENQUEUE. The experiment's status becomes Pending. When the worker fetches the experiment from the queue, the status becomes Running, and you can view its progress in the info panel. When the status becomes Completed, go to the next step.
Step 6. Compare the experiments
To compare the original and tuned experiments, do the following:
- In the Trains Web-App (UI), on the Projects page, click the examples project.
- In the experiments table, select the checkboxes for our two experiments (
Tensorflow mnist with summaries exampleand
Clone Of Tensorflow v2 mnist with summaries).
On the menu bar at the bottom of the experiment table, click COMPARE. The experiment comparison window appears. All differences appear with a different background color to highlight them.
The experiment comparison window is organized in the following tabs:
- DETAILS - The ARTIFACTS section, including input and output models with their network designs, and other artifacts; the EXECUTION section execution, including source code control, installed Python packages and versions, uncommitted changes, and the Docker image name which, in this case, is empty.
- HYPER PARAMETERS - The hyperparameters and their values.
- SCALARS - Scalar metrics with the option to view them as charts or values.
- PLOTS - Plots of any data with the option to view them as charts or values.
- DEBUG SAMPLES - Media including images, audio, and video uploaded by your experiment shown as thumbnails.
Examine the differences in our two experiments by doing the following:
- Click the DETAILS > EXECUTION > Parameters. The hyperparameters dropout and learning_rate are shown with a different background color. The values are different.
- Click the SCALARS > To the right of Add Experiment, click Values. The scalar values appear as values showing the difference highlighted by background color.
- To the right of Add Experiment, click Chart. The scalar charts appear showing the difference.
- For more information about all editable parts of an experiment, see Modify experiments for tuning in the User Interface section.