Explicit Reporting

In Trains, explicit reporting extends automatic logging. Explicit reporting includes:

  • Logging text messages, scalars, plots in several formats, tables, images, audio, and video. In addition to the examples on this page, see logger.py and manual_reporting.py in the trains repository, and the Logger Module reference page.

    The debug sample storage destination

    Trains allows you to set a default storage location for debug samples, and another default storage location for model snapshots and artifacts. In Trains, these are different locations you can use to manage your storage more efficiently.

  • Parameter dictionaries whose parameters are tracked with the experiment.
  • Environment variables which you can specify for tracking.

Pre-populated examples ready to enqueue

A self-hosted Trains Server installs with the example experiments located in the trains repository, examples folder

Plots

Scalars

To plot scalars, use the Logger.report_scalar method.

logger.report_scalar('example scalar', 'series A', iteration=0, value=25)
logger.report_scalar('example scalar', 'series A', iteration=1, value=75)
logger.report_scalar('example scalar', 'series A', iteration=4, value=100)

Trains reports scalar plots in the experiment results SCALARS tab.

Histograms

To plot a data series as a histogram, use Logger.report_histogram. Set the mode parameter for the type of histogram. The mode values are group (the default), stack, and relative.

A histogram using the default mode of group:

histogram_A = np.random.randint(10, size=10)

histogram_B = np.random.randint(10, size=10)

logger.report_histogram("example histogram group", "group A", iteration=0,
    values=histogram_A, xaxis='title x', yaxis='title y')

logger.report_histogram("example histogram group", "group B", iteration=0,
    values=histogram_B, xaxis='title x', yaxis='title y')

Trains reports histograms in the experiment results PLOTS tab.

A histogram setting mode to stack:

logger.report_histogram("example histogram stack", "stack A", iteration=1,
    values=histogram_A, xaxis='title x', yaxis='title y', mode='stack')

logger.report_histogram("example histogram stack", "stack B", iteration=1,
    values=histogram_B, xaxis='title x', yaxis='title y', mode='stack')

A histogram setting mode to relative:

logger.report_histogram("example histogram relative", "relative A", iteration=2,
    values=histogram_A, xaxis='title x', yaxis='title y', mode='relative')

logger.report_histogram("example histogram relative", "relative B", iteration=2,
    values=histogram_B, xaxis='title x', yaxis='title y', mode='relative')

Line plots

To plot a series as a line, data points as markers, or both lines and markers, use the Logger.report_line_plot method.

In this example, call Logger.report_line_plot to report one series whose data is a 2D Numpy array.

name = 'Series Data'
data = np.hstack((np.atleast_2d(np.arange(0, 10)).T, np.random.randint(10, size=(10, 1))))
labels = ['Label %2d' % x for x in range(10)]
seriesinfo = [{'name': name, 'data': data, 'labels': labels}]

logger.report_line_plot(title='example line plot', series=seriesinfo, iteration=1,
    xaxis='title x', yaxis='title y', mode='lines+markers')

Trains reports line plots in the experiment results PLOTS tab.

Plot more than one series on the same plot by specifying the same iteration and title parameter arguments, with different series arguments for each call.

In this example, call Logger.report_line_plot to report two series whose data is two, 2D Numpy arrays.

data_A = np.hstack((np.atleast_2d(np.arange(0, 10)).T, np.random.randint(10, size=(10, 1))))
data_B = np.hstack((np.atleast_2d(np.arange(0, 10)).T, np.random.randint(10, size=(10, 1))))
seriesinfo = [{'name':'Series One', 'data': data_A, 'labels': labels},
              {'name': 'Series Two', 'data': data_B, 'labels':labels}]

logger.report_line_plot(title='example line plot (multiple)', series=seriesinfo,
                        iteration=1, xaxis='title x', yaxis='title y')

2D scatter plots

To plot a series as a 2-dimensional scatter plot, use the Logger.report_scatter2d method.

To plot a 2-dimensional scatter plot as markers for the data points, specify the mode parameter argument as markers.

# report scatter 2D
scatter2d = np.hstack((np.atleast_2d(np.arange(0, 10)).T,
    np.random.randint(10, size=(10, 1))))

logger.report_scatter2d("example 2D scatter markers", "series_1", iteration=1,
    scatter=scatter2d, xaxis='title x', yaxis='title y', mode='markers')

Trains reports 2-D scatter plots in the experiment results PLOTS tab.

To plot lines, specify the mode parameter argument as lines.



logger.report_scatter2d("example 2D scatter lines", "series_1", iteration=1,
    scatter=scatter2d, xaxis='title x', yaxis='title y', mode='lines')

To plot lines and markers, specify the mode parameter argument as lines+markers.

logger.report_scatter2d("example 2D scatter markers and lines", "series_1", iteration=1,
    scatter=scatter2d, xaxis='title x', yaxis='title y', mode='lines+markers')

Plot more than one series on the same plot by specifying the same iteration and title parameter arguments, with different series arguments for each call.

# report scatter 2D (multiple)
scatter2d_1 = np.hstack((np.atleast_2d(np.arange(0, 10)).T,
    np.random.randint(10, size=(10, 1))))
logger.report_scatter2d("example 2D scatter (multiple)", "series_1", iteration=1,
    scatter=scatter2d_1, xaxis='title x', yaxis='title y', mode='markers')

scatter2d_2 = np.hstack((np.atleast_2d(np.arange(0, 10)).T,
    np.random.randint(10, size=(10, 1))))
logger.report_scatter2d("example 2D scatter (multiple)", "series_2", iteration=1,
    scatter=scatter2d_2, xaxis='title x', yaxis='title y', mode='markers')

3D scatter plots

To plot a series as a 3-dimensional scatter plot, use the Logger.report_scatter3d method.

# report 3d scatter plot
scatter3d = np.random.randint(10, size=(10, 3))
logger.report_scatter3d("example 3D scatter", "series_xyz", iteration=1, scatter=scatter3d,
                        xaxis="title x", yaxis="title y", zaxis="title z")

Trains reports 3-D scatter plots in the experiment results PLOTS tab.

Confusion matrices

To plot a series as a confusion matrix (heat-map matrix), use the Logger.report_confusion_matrix method

# report confusion matrix
confusion = np.random.randint(10, size=(10, 10))
logger.report_matrix("example confusion matrix", "ignored", iteration=1,
    matrix=confusion, xaxis="title X", yaxis="title Y")

Trains reports confusion matrices in the experiment results PLOTS tab.

Surface plots

To plot a series as a surface plot, use the Logger.report_surface method.

# report 3d surface
logger.report_surface("example surface plot", "series1", iteration=1,
    matrix=confusion, xaxis="title X", yaxis="title Y", zaxis="title Z")

Trains reports surface plots in the experiment results HYPER PARAMETERS tab.

Images (Numpy arrays, PIL images, file location)

To report an image and upload its contents to the bucket specified in the Trains configuration file, or a default upload destination which can be a bucket or folder location, use Logger.report_image method.

Also, report images using Logger.report_media method which allows you to report images in other formats, see Images in the "Media" section of this page.

Image formats include:

  • Numpy arrays
  • PIL images
  • images specified by URL
  • images specified by file location
# reporting images
m = np.eye(256, 256, dtype=np.float)
logger.report_image('example images', "image float", iteration=1, image=m)
m = np.eye(256, 256, dtype=np.uint8)*255
logger.report_image('example images', "image uint8", iteration=1, image=m)
m = np.concatenate((np.atleast_3d(m), np.zeros((256, 256, 2), dtype=np.uint8)), axis=2)
logger.report_image('example images', "image color red", iteration=1, image=m)
image_open = Image.open(os.path.join("samples", "picasso.jpg"))
logger.report_image('example images', "image PIL", iteration=1, image=image_open)

Trains reports images in the experiment results DEBUG SAMPLES tab.

Tables

Report Pandas DataFrames and CSV files as table.

Pandas DataFrame

To report a Pandas DataFrame, use the Logger.report_table method, specifying the Pandas DataFrame as the table_plot argument.

# reporting tables - a Pandas DataFrame
df = pd.DataFrame(
        {
            'num_legs': [2, 4, 8, 0],
            'num_wings': [2, 0, 0, 0],
            'num_specimen_seen': [10, 2, 1, 8]
        },
        index=['falcon', 'dog', 'spider', 'fish']
    )
    logger.report_table("example table Pandas dataframe", "PD with index", 1, table_plot=df)

Trains reports tables in the experiment results PLOTS tab.

CSV files

To report the contents of a CSV file as a table, use the Logger.report_table method, specifying the CSV URL as the url parameter argument, or a path to a local CSV file as the csv parameter argument.

# Report table - CSV from path
    csv_url = "https://raw.githubusercontent.com/plotly/datasets/master/Mining-BTC-180.csv"
    logger.report_table("example table CVS", "remote csv", 1, url=csv_url)

Trains reports tables in the experiment results PLOTS tab.

Media (images, audio, video)

To report media including images, audio, and video, and upload its contents , use Logger.report_media method. You can upload the media to the bucket specified in the Trains configuration file, or a default upload destination which can be a bucket or folder location.

Trains reports media in the experiment results DEBUG SAMPLES tab.

When you call the Logger.report_media method, specify the media using one of the following parameters:

  • local_path - image file
  • stream and streamfile_extension - BytesIO stream
  • url - a URL for the location of a pre-uploaded image

Also, report images using Logger.report_image method which allows you to report images in other formats, see Images (Numpy arrays, PIL images, file location) on this page.

Images

# report an image
logger.report_media(title='example_media_image', series='image', iteration=0, local_path='../code_other/samples/picasso.jpg')

  • Open the image viewer by clicking the image thumbnail.

Audio

# report audio
logger.report_media(title='example_media_audio', series='audio', iteration=1, local_path='../code_other/samples/sample.mp3')

  • Open the audio player by clicking the audio thumbnail.

Video

# report video
logger.report_media(title='example_media_video', series='video', iteration=2, local_path='../code_other/samples/sample.mp4')

  • Open the video player by clicking the video thumbnail.

Hyper-parameters and environment variables

Logging experiment parameter dictionaries

To log a dictionary of parameters, use the Task.connect method. After connecting the dictionary, Trains updates changes to it for tracking with the experiment.

For example, to log the hyperparameters learning_rate, batch_size, display_step, model_path, n_hidden_1, and n_hidden_2:

# Create a dictionary of parameters
parameters_dict = { 'learning_rate': 0.001, 'batch_size': 100, 'display_step': 1, 
    'model_path': "/tmp/model.ckpt", 'n_hidden_1': 256, 'n_hidden_2': 256 }

# Connect the dictionary to your TRAINS Task    
parameters_dict = Task.current_task().connect(parameters_dict)

Later in the experiment, change the parameter dictionary.

parameters_dict['display_step'] = 999
parameters_dict['new_parameter'] = 'new value'

Trains reports parameters in the experiment results HYPER PARAMETERS tab.

Specifying environment variables to track

To log environment variables for tracking, use the TRAINS_LOG_ENVIRONMENT environment variable as follows:

  • All environment variables

    export TRAINS_LOG_ENVIRONMENT="*"
    
  • Specific environment variables

    For example, log PWD and PYTHONPATH

    export TRAINS_LOG_ENVIRONMENT="PWD,PYTHONPATH"
    
  • No environment variables

    export TRAINS_LOG_ENVIRONMENT=
    

Console and logger messages

To log to the console and print to the log, use the Logger.report_text method.

logger.report_text("Log this to the console and log.")

Trains reports messages in the experiment results LOGS tab and to the console, unless the print_console parameter argument is False.

Optionally, disable console logging.

logger.report_text("Log this, but not to the console.", print_console=False)

Logger object and storage methods

Get the current logger

Before calling Logger report methods, you must get a logger object. Do not construction a Logger object directly. By using Logger.get_logger, Trains connects all explicit reporting with the Task (experiment).

In this example, we call the Task.current_task method to connect the Logger object to the currently running experiment.

# get logger object for any metrics / reports
logger = Task.current_task().get_logger()

Set a default upload destination

To specify a default, destination storage location for media, use the Logger.set_default_upload_destination method. Trains reports a link to the uploaded media.

The destinations include S3, Google Cloud Storage, and a file path. Credentials for the storage location are in the Trains configuration file.

An upload creates the following folder structure in the destination location:

+-- <project-name>
|   +-- <experiment-name>.<Task-Id>
|       +-- metrics
|           +-- <title>
|               +-- <series>
|                   +-- <media-name>

For example, set a folder as the destination:

# set default upload destination
logger.set_default_upload_destination('/mnt/data/media')

Trains storage destinations

In Trains, you can use different storage destinations for different purposes, including the default upload destination for media, and the storage output URI destination for model snapshots and artifacts.

You can set the storage destination for checkpointing models in any of the following ways: