Table Of Contents

  1. Docs
  2. Model Module

Model Module

class trains.model.InputModel

Load an existing model in the system, search by model id. The Model will be read-only and can be used to pre initialize a network We can connect the model to a task as input model, then when running remotely override it with the UI.

Parameters

model_id (str) – The Trains Id (system UUID) of the input model whose metadata the Trains Server (backend) stores.

classmethod import_model(weights_url, config_text=None, config_dict=None, label_enumeration=None, name=None, tags=None, comment=None, is_package=False, create_as_published=False, framework=None)

Create an InputModel object from a pre-trained model by specifying the URL of an initial weight files. Optionally, input a configuration, label enumeration, name for the model, tags describing the model, comment as a description of the model, indicate whether the model is a package, specify the model’s framework, and indicate whether to immediately set the model’s status to Published. The model is read-only.

The Trains Server (backend) may already store the model’s URL. If the input model’s URL is not stored, meaning the model is new, then it is imported and Trains stores its metadata. If the URL is already stored, the import process stops, Trains issues a warning message, and Trains reuses the model.

In your Python experiment script, after importing the model, you can connect it to the main execution Task as an input model using InputModel.connect() or Task.connect(). That initializes the network.

Note

Using the Trains Web-App (user interface), you can reuse imported models and switch models in experiments.

Parameters

  • weights_url (str) –

    A valid URL for the initial weights file. If the Trains Web-App (backend) already stores the metadata of a model with the same URL, that existing model is returned and Trains ignores all other parameters.

    For example:

    • https://domain.com/file.bin

    • s3://bucket/file.bin

    • file:///home/user/file.bin

  • config_text (unconstrained text string) – The configuration as a string. This is usually the content of a configuration dictionary file. Specify config_text or config_dict, but not both.

  • config_dict (dict) – The configuration as a dictionary. Specify config_text or config_dict, but not both.

  • label_enumeration (dict) –

    Optional label enumeration dictionary of string (label) to integer (value) pairs.

    For example:

    {
     'background': 0,
     'person': 1
}

  • name (str) – The name of the newly imported model. (Optional)

  • tags (list(str)) – The list of tags which describe the model. (Optional)

  • comment (str) – A comment / description for the model. (Optional)

  • is_package (bool) –

    Is the imported weights file is a package? (Optional)

    • True - Is a package. Add a package tag to the model.

    • False - Is not a package. Do not add a package tag. (Default)

  • create_as_published (bool) –

    Set the model’s status to Published? (Optional)

    • True - Set the status to Published.

    • False - Do not set the status to Published. The status will be Draft. (Default)

  • framework (str or Framework object) – The framework of the model. (Optional)

Returns

The imported model or existing model (see above).

classmethod load_model(weights_url, load_archived=False)

Load an already registered model based on a pre-existing model file (link must be valid). If the url to the weights file already exists, the returned object is a Model representing the loaded Model. If no registered model with the specified url is found, None is returned.

Parameters

  • weights_url –

    The valid url for the weights file (string).

    Examples:

    "https://domain.com/file.bin" or "s3://bucket/file.bin" or "file:///home/user/file.bin".

    Note

    If a model with the exact same URL exists, it will be used, and all other arguments will be ignored.

  • load_archived (bool) –

    Load archived models?

    • True - Load the registered Model, if it is archived.

    • False - Ignore archive models.

Returns

The InputModel object, or None if no model could be found.

classmethod empty(config_text=None, config_dict=None, label_enumeration=None)

Create an empty model object. Later, you can assign a model to the empty model object.

Parameters

  • config_text (unconstrained text string) – The model configuration as a string. This is usually the content of a configuration dictionary file. Specify config_text or config_dict, but not both.

  • config_dict (dict) – The model configuration as a dictionary. Specify config_text or config_dict, but not both.

  • label_enumeration (dict) –

    The label enumeration dictionary of string (label) to integer (value) pairs. (Optional)

    For example:

    {
     'background': 0,
     'person': 1
}

Returns

An empty model object.

property id

The Id (system UUID) of the model.

Returns

The model ID.

connect(task)

Connect the current model to a Task object, if the model is preexisting. Preexisting models include:

  • Imported models (InputModel objects created using the Logger.import_model() method).

  • Models whose metadata is already in the Trains platform, meaning the InputModel object is instantiated from the InputModel class specifying the the model’s Trains Id as an argument.

  • Models whose origin is not Trains that are used to create an InputModel object. For example, models created using TensorFlow models.

When the experiment is executed remotely in a worker, the input model already specified in the experiment is used.

Note

The Trains Web-App allows you to switch one input model for another and then enqueue the experiment to execute in a worker.

Parameters

task (object) – A Task object.

property comment

The comment for the model. Also, use for a model description.

Returns

The model comment / description.

property config_dict

The configuration as a dictionary, parsed from the design text. This usually represents the model configuration. For example, prototxt, an ini file, or Python code to evaluate.

Returns

The configuration.

property config_text

The configuration as a string. For example, prototxt, an ini file, or Python code to evaluate.

Returns

The configuration.

get_local_copy(extract_archive=True, raise_on_error=False)

Retrieve a valid link to the model file(s). If the model URL is a file system link, it will be returned directly. If the model URL is points to a remote location (http/s3/gs etc.), it will download the file(s) and return the temporary location of the downloaded model.

Parameters

  • extract_archive (bool) – If True and the model is of type ‘packaged’ (e.g. TensorFlow compressed folder) The returned path will be a temporary folder containing the archive content

  • raise_on_error (bool) – If True and the artifact could not be downloaded, raise ValueError, otherwise return None on failure and output log warning.

Returns

A local path to the model (or a downloaded copy of it).

get_weights(raise_on_error=False)

Download the base model and return the locally stored filename.

Parameters

raise_on_error (bool) – If True and the artifact could not be downloaded, raise ValueError, otherwise return None on failure and output log warning.

Returns

The locally stored file.

get_weights_package(return_path=False, raise_on_error=False)

Download the base model package into a temporary directory (extract the files), or return a list of the locally stored filenames.

Parameters

  • return_path (bool) –

    Return the model weights or a list of filenames? (Optional)

    • True - Download the model weights into a temporary directory, and return the temporary directory path.

    • False - Return a list of the locally stored filenames. (Default)

  • raise_on_error (bool) – If True and the artifact could not be downloaded, raise ValueError, otherwise return None on failure and output log warning.

Returns

The model weights, or a list of the locally stored filenames.

property labels

The label enumeration of string (label) to integer (value) pairs.

Returns

A dictionary containing labels enumeration, where the keys are labels and the values as integers.

property name

The name of the model.

Returns

The model name.

publish()

Set the model to the status published and for public use. If the model’s status is already published, then this method is a no-op.

property tags

A list of tags describing the model.

Returns

The list of tags.

property task

Return the creating task id (str)

Returns

The Task ID.

property url

Return the url of the model file (or archived files)

Returns

The model file URL.

class trains.model.OutputModel

Create an output model for a Task (experiment) to store the training results.

The OutputModel object is always connected to a Task object, because it is instantiated with a Task object as an argument. It is, therefore, automatically registered as the Task’s (experiment’s) output model.

The OutputModel object is read-write.

A common use case is to reuse the OutputModel object, and override the weights after storing a model snapshot. Another use case is to create multiple OutputModel objects for a Task (experiment), and after a new high score is found, store a model snapshot.

If the model configuration and / or the model’s label enumeration are None, then the output model is initialized with the values from the Task object’s input model.

Note

When executing a Task (experiment) remotely in a worker, you can modify the model configuration and / or model’s label enumeration using the Trains Web-App.

Create a new model and immediately connect it to a task.

We do not allow for Model creation without a task, so we always keep track on how we created the models In remote execution, Model parameters can be overridden by the Task (such as model configuration & label enumerator)

Parameters

  • task (Task) – The Task object with which the OutputModel object is associated.

  • config_text (unconstrained text string) – The configuration as a string. This is usually the content of a configuration dictionary file. Specify config_text or config_dict, but not both.

  • config_dict (dict) – The configuration as a dictionary. Specify config_dict or config_text, but not both.

  • label_enumeration (dict) –

    The label enumeration dictionary of string (label) to integer (value) pairs. (Optional)

    For example:

    {
     'background': 0,
     'person': 1
}

  • name (str) – The name for the newly created model. (Optional)

  • tags (list(str)) – A list of strings which are tags for the model. (Optional)

  • comment (str) – A comment / description for the model. (Optional)

  • framework (str or Framework object) – The framework of the model or a Framework object. (Optional)

  • base_model_id – optional, model id to be reused

property published

Get the published state of this model.

Returns

property config_text

Get the configuration as a string. For example, prototxt, an ini file, or Python code to evaluate.

Returns

The configuration.

property config_dict

Get the configuration as a dictionary parsed from the config_text text. This usually represents the model configuration. For example, from prototxt to ini file or python code to evaluate.

Returns

The configuration.

property labels

Get the label enumeration as a dictionary of string (label) to integer (value) pairs.

For example:

{
     'background': 0,
     'person': 1
}
Returns

The label enumeration.

connect(task)

Connect the current model to a Task object, if the model is a preexisting model. Preexisting models include:

  • Imported models.

  • Models whose metadata the Trains Server (backend) is already storing.

  • Models from another source, such as frameworks like TensorFlow.

Parameters

task (object) – A Task object.

set_upload_destination(uri)

Set the URI of the storage destination for uploaded model weight files. Supported storage destinations include S3, Google Cloud Storage), and file locations.

Using this method, files uploads are separate and then a link to each is stored in the model object.

Note

For storage requiring credentials, the credentials are stored in the Trains configuration file, ~/trains.conf.

Parameters

uri (str) –

The URI of the upload storage destination.

For example:

  • s3://bucket/directory/

  • file:///tmp/debug/

Return bool

The status of whether the storage destination schema is supported.

  • True - The storage destination scheme is supported.

  • False - The storage destination scheme is not supported.

update_weights(weights_filename=None, upload_uri=None, target_filename=None, auto_delete_file=True, register_uri=None, iteration=None, update_comment=True)

Update the model weights from a locally stored model filename.

Note

Uploading the model is a background process. A call to this method returns immediately.

Parameters

  • weights_filename (str) – The name of the locally stored weights file to upload. Specify weights_filename or register_uri, but not both.

  • upload_uri (str) – The URI of the storage destination for model weights upload. The default value is the previously used URI. (Optional)

  • target_filename (str) – The newly created filename in the storage destination location. The default value is the weights_filename value. (Optional)

  • auto_delete_file (bool) –

    Delete the temporary file after uploading? (Optional)

    • True - Delete (Default)

    • False - Do not delete

  • register_uri (str) – The URI of an already uploaded weights file. The URI must be valid. Specify register_uri or weights_filename, but not both.

  • iteration (int) – The iteration number.

  • update_comment (bool) –

    Update the model comment with the local weights file name (to maintain provenance)? (Optional)

    • True - Update model comment (Default)

    • False - Do not update

Returns

The uploaded URI.

update_weights_package(weights_filenames=None, weights_path=None, upload_uri=None, target_filename=None, auto_delete_file=True, iteration=None)

Update the model weights from locally stored model files, or from directory containing multiple files.

Note

Uploading the model weights is a background process. A call to this method returns immediately.

Parameters

  • weights_filenames (list(str)) – The file names of the locally stored model files. Specify weights_filenames, or weights_path, but not both.

  • weights_path (str) – The directory path to a package. All the files in the directory will be uploaded. Specify weights_path or weights_filenames, but not both.

  • upload_uri (str) – The URI of the storage destination for the model weights upload. The default is the previously used URI. (Optional)

  • target_filename (str) – The newly created filename in the storage destination URI location. The default is the value specified in the weights_filename parameter. (Optional)

  • auto_delete_file (bool) –

    Delete temporary file after uploading? (Optional)

    • True - Delete (Default)

    • False - Do not delete

  • iteration (int) – The iteration number.

Returns

The uploaded URI for the weights package.

update_design(config_text=None, config_dict=None)

Update the model configuration. Store a blob of text for custom usage.

Note

This method’s behavior is lazy. The design update is only forced when the weights are updated.

Parameters

  • config_text (unconstrained text string) – The configuration as a string. This is usually the content of a configuration dictionary file. Specify config_text or config_dict, but not both.

  • config_dict (dict) – The configuration as a dictionary. Specify config_text or config_dict, but not both.

Returns

True, update successful. False, update not successful.

property comment

The comment for the model. Also, use for a model description.

Returns

The model comment / description.

get_weights(raise_on_error=False)

Download the base model and return the locally stored filename.

Parameters

raise_on_error (bool) – If True and the artifact could not be downloaded, raise ValueError, otherwise return None on failure and output log warning.

Returns

The locally stored file.

get_weights_package(return_path=False, raise_on_error=False)

Download the base model package into a temporary directory (extract the files), or return a list of the locally stored filenames.

Parameters

  • return_path (bool) –

    Return the model weights or a list of filenames? (Optional)

    • True - Download the model weights into a temporary directory, and return the temporary directory path.

    • False - Return a list of the locally stored filenames. (Default)

  • raise_on_error (bool) – If True and the artifact could not be downloaded, raise ValueError, otherwise return None on failure and output log warning.

Returns

The model weights, or a list of the locally stored filenames.

property id

The Id (system UUID) of the model.

Returns

The model ID.

property name

The name of the model.

Returns

The model name.

publish()

Set the model to the status published and for public use. If the model’s status is already published, then this method is a no-op.

property tags

A list of tags describing the model.

Returns

The list of tags.

property task

Return the creating task id (str)

Returns

The Task ID.

update_labels(labels)

Update the label enumeration.

Parameters

labels (dict) –

The label enumeration dictionary of string (label) to integer (value) pairs.

For example:

{
     'background': 0,
     'person': 1
}

Returns

property url

Return the url of the model file (or archived files)

Returns

The model file URL.

classmethod wait_for_uploads(timeout=None, max_num_uploads=None)

Wait for any pending or in-progress model uploads to complete. If no uploads are pending or in-progress, then the wait_for_uploads returns immediately.

Parameters

  • timeout (float) – The timeout interval to wait for uploads (seconds). (Optional).

  • max_num_uploads (int) – The maximum number of uploads to wait for. (Optional).

class trains.model.Model

Represent an existing model in the system, search by model id. The Model will be read-only and can be used to pre initialize a network

Load model based on id, returned object is read-only and can be connected to a task

Notice, we can override the input model when running remotely

Parameters

model_id – id (string)

get_local_copy(extract_archive=True, raise_on_error=False)

Retrieve a valid link to the model file(s). If the model URL is a file system link, it will be returned directly. If the model URL is points to a remote location (http/s3/gs etc.), it will download the file(s) and return the temporary location of the downloaded model.

Parameters

  • extract_archive (bool) – If True and the model is of type ‘packaged’ (e.g. TensorFlow compressed folder) The returned path will be a temporary folder containing the archive content

  • raise_on_error (bool) – If True and the artifact could not be downloaded, raise ValueError, otherwise return None on failure and output log warning.

Returns

A local path to the model (or a downloaded copy of it).

property comment

The comment for the model. Also, use for a model description.

Returns

The model comment / description.

property config_dict

The configuration as a dictionary, parsed from the design text. This usually represents the model configuration. For example, prototxt, an ini file, or Python code to evaluate.

Returns

The configuration.

property config_text

The configuration as a string. For example, prototxt, an ini file, or Python code to evaluate.

Returns

The configuration.

get_weights(raise_on_error=False)

Download the base model and return the locally stored filename.

Parameters

raise_on_error (bool) – If True and the artifact could not be downloaded, raise ValueError, otherwise return None on failure and output log warning.

Returns

The locally stored file.

get_weights_package(return_path=False, raise_on_error=False)

Download the base model package into a temporary directory (extract the files), or return a list of the locally stored filenames.

Parameters

  • return_path (bool) –

    Return the model weights or a list of filenames? (Optional)

    • True - Download the model weights into a temporary directory, and return the temporary directory path.

    • False - Return a list of the locally stored filenames. (Default)

  • raise_on_error (bool) – If True and the artifact could not be downloaded, raise ValueError, otherwise return None on failure and output log warning.

Returns

The model weights, or a list of the locally stored filenames.

property id

The Id (system UUID) of the model.

Returns

The model ID.

property labels

The label enumeration of string (label) to integer (value) pairs.

Returns

A dictionary containing labels enumeration, where the keys are labels and the values as integers.

property name

The name of the model.

Returns

The model name.

publish()

Set the model to the status published and for public use. If the model’s status is already published, then this method is a no-op.

property tags

A list of tags describing the model.

Returns

The list of tags.

property task

Return the creating task id (str)

Returns

The Task ID.

property url

Return the url of the model file (or archived files)

Returns

The model file URL.

Logger Module
Storage Module
© 2020 • Allegro AI. All Rights Reserved.