trains.model

class trains.model.InputModel

Load an existing model in the system, search by model id. The Model will be read-only and can be used to pre initialize a network We can connect the model to a task as input model, then when running remotely override it with the UI.

Load model based on id, returned object is read-only and can be connected to a task

Notice, we can override the input model when running remotely

Parameters

model_id – id (string)

classmethod import_model(weights_url, config_text=None, config_dict=None, label_enumeration=None, name=None, tags=None, comment=None, is_package=False, create_as_published=False, framework=None)

Create a model from pre-existing model file (link must be valid), and model configuration.

If the url to the weights file already exists, the import process will stop with a warning and automatically it will try to import the model that was found. The Model will be read-only and can be used to pre initialize a network We can connect the model to a task as input model, then when running remotely override it with the UI. Load model based on id, returned object is read-only and can be connected to a task That is, we can override the input model when running remotely

Parameters
  • weights_url – valid url for the weights file (string). examples: “https://domain.com/file.bin” or “s3://bucket/file.bin” or “file:///home/user/file.bin”. NOTE: if a model with the exact same URL exists, it will be used, and all other arguments will be ignored.

  • config_text – model configuration (unconstrained text string). usually the content of configuration file. If config_text is not None, config_dict must not be provided.

  • config_dict – model configuration parameters (dict). If config_dict is not None, config_text must not be provided.

  • label_enumeration – dictionary of string to integer, enumerating the model output to labels example: {‘background’: 0 , ‘person’: 1}

  • name – optional, name for the newly imported model

  • tags – optional, list of strings as tags

  • comment – optional, string description for the model

  • is_package – Boolean. Indicates that the imported weights file is a package. If True, and a new model was created, a package tag will be added.

  • create_as_published – Boolean. If True, and a new model is created, it will be published.

  • framework – optional, string name of the framework of the model or Framework

classmethod empty(config_text=None, config_dict=None, label_enumeration=None)

Create an empty model, so that later we can execute the task in remote and replace the empty model with pre-trained model file

Parameters
  • config_text – model configuration (unconstrained text string). usually the content of a config_dict file. If config_text is not None, config_dict must not be provided.

  • config_dict – model configuration parameters (dict). If config_dict is not None, config_text must not be provided.

  • label_enumeration – dictionary of string to integer, enumerating the model output to labels example: {‘background’: 0 , ‘person’: 1}

property id

return the id of the model (string)

Returns

model id (string)

connect(task)

Connect current model with a specific task, only supported for preexisting models,

i.e. not supported on objects created with create_and_connect() When running in debug mode (i.e. locally), the task is updated with the model object (i.e. task input model is the load_model_id) When running remotely (i.e. from a daemon) the model is being updated from the task Notice! when running remotely the load_model_id is ignored and loaded from the task object regardless of the code

Parameters

task – Task object

property comment

return comment/description of the model (string)

Returns

model description (string)

property config_dict

returns a configuration dictionary parsed from the design text, usually representing the model configuration (from prototxt to ini file or python code to evaluate)

Returns

Dictionary

property config_text

returns a string representing the model configuration (from prototxt to ini file or python code to evaluate)

Returns

string

get_weights()

Download the base model and returns a string of locally stored filename

Returns

string to locally stored file

get_weights_package(return_path=False)

Download the base model package, extract the files and return list of locally stored filenames

Parameters

return_path – if True the model weights are downloaded into a temporary directory and the directory path is returned, instead of list of files

Returns

string to locally stored file

property labels

Return the labels enumerator {str(label): integer(id)} as saved in the model object

Returns

labels_dict, dictionary with labels (text) keys and values as integers

property name

return the name of the model (string)

Returns

model name (string)

publish()

Set the model to ‘published’ and set it for public use.

If the model is already published, this method is a no-op.

property tags

Return the list of tags the model has

Returns

list of strings (tags)

class trains.model.OutputModel

Create an output model for a task to store the training results in.

By definition the Model is always connected to a task, and is automatically registered as its output model. The common use case is reusing the model object, and overriding the weights every stored snapshot. A user can create multiple output models for a task, think a snapshot after a validation test has a new high-score. The Model will be read-write and if config/label-enumeration are None, their values will be initialized from the task input model.

Create a new model and immediately connect it to a task.

We do not allow for Model creation without a task, so we always keep track on how we created the models In remote execution, Model parameters can be overridden by the Task (such as model configuration & label enumerator)

Parameters
  • task (Task) – Task object

  • config_text – model configuration (unconstrained text string). usually the content of a config_dict file. If config_text is not None, config_dict must not be provided.

  • config_dict – model configuration parameters (dict). If config_dict is not None, config_text must not be provided.

  • label_enumeration (dict[str: int] or None) – dictionary of string to integer, enumerating the model output to labels example: {‘background’: 0 , ‘person’: 1}

  • name – optional, name for the newly created model

  • tags – optional, list of strings as tags

  • comment – optional, string description for the model

  • framework – optional, string name of the framework of the model or Framework

property config_text

returns a string representing the model configuration (from prototxt to ini file or python code to evaluate)

Returns

string

property config_dict

returns a configuration dictionary parsed from the config_text text, usually representing the model configuration (from prototxt to ini file or python code to evaluate)

Returns

Dictionary

property labels

Return the labels enumerator {str(label): integer(id)} as saved in the model object

Returns

labels_dict, dictionary with labels (text) keys and values as integers

connect(task)

Connect current model with a specific task, only supported for preexisting models,

i.e. not supported on objects created with create_and_connect() When running in debug mode (i.e. locally), the task is updated with the model object (i.e. task input model is the load_model_id) When running remotely (i.e. from a daemon) the model is being updated from the task Notice! when running remotely the load_model_id is ignored and loaded from the task object regardless of the code

Parameters

task – Task object

set_upload_destination(uri)

Set the uri to upload all the model weight files to.

Files are uploaded separately to the destination storage (e.g. s3,gc,file) and then a link to the uploaded model is stored in the model object Notice: credentials for the upload destination will be pooled from the global configuration file (i.e. ~/trains.conf)

Parameters

uri – upload destination (string). example: ‘s3://bucket/directory/’ or ‘file:///tmp/debug/’

Returns

True if destination scheme is supported (i.e. s3:// file:// gc:// etc…)

update_weights(weights_filename=None, upload_uri=None, target_filename=None, auto_delete_file=True, register_uri=None, iteration=None, update_comment=True)

Update the model weights from a locally stored model filename.

Uploading the model is a background process, the call returns immediately.

Parameters
  • weights_filename – locally stored filename to be uploaded as is

  • upload_uri – destination uri for model weights upload (default: previously used uri)

  • target_filename – the newly created filename in the destination uri location (default: weights_filename)

  • auto_delete_file – delete temporary file after uploading

  • register_uri – register an already uploaded weights file (uri must be valid)

  • update_comment – if True, model comment will be updated with local weights file location (provenance)

Returns

uploaded uri

update_weights_package(weights_filenames=None, weights_path=None, upload_uri=None, target_filename=None, auto_delete_file=True, iteration=None)

Update the model weights from a locally stored model files (or directory containing multiple files).

Uploading the model is a background process, the call returns immediately.

Parameters
  • weights_filenames (list) – list of locally stored filenames (list of strings)

  • weights_path (str) – directory path to package (all the files in the directory will be uploaded)

  • upload_uri – destination uri for model weights upload (default: previously used uri)

  • target_filename – the newly created filename in the destination uri location (default: weights_filename)

  • auto_delete_file – delete temporary file after uploading

Returns

uploaded uri for the weights package

update_design(config_text=None, config_dict=None)

Update the model configuration, basically store a blob of text for custom usage

Notice: this is done in a lazily, only when updating weights we force the update of configuration in the backend

Parameters
  • config_text – model configuration (unconstrained text string). usually the content of a config_dict file. If config_text is not None, config_dict must not be provided.

  • config_dict – model configuration parameters (dict). If config_dict is not None, config_text must not be provided.

Returns

True if update was successful

update_labels(labels)

Update the model label enumeration {str(label): integer(id)}

Parameters

labels – dictionary with labels (text) keys and values as integers example: {‘background’: 0 , ‘person’: 1}

Returns

classmethod wait_for_uploads(timeout=None, max_num_uploads=None)

Wait for any pending/in-progress model uploads. If no uploads are pending or in-progress, returns immediately.

Parameters
  • timeout – If not None, a floating point number specifying a timeout in seconds after which this call will return.

  • max_num_uploads – Max number of uploads to wait for.

property comment

return comment/description of the model (string)

Returns

model description (string)

get_weights()

Download the base model and returns a string of locally stored filename

Returns

string to locally stored file

get_weights_package(return_path=False)

Download the base model package, extract the files and return list of locally stored filenames

Parameters

return_path – if True the model weights are downloaded into a temporary directory and the directory path is returned, instead of list of files

Returns

string to locally stored file

property id

return the id of the model (string)

Returns

model id (string)

property name

return the name of the model (string)

Returns

model name (string)

publish()

Set the model to ‘published’ and set it for public use.

If the model is already published, this method is a no-op.

property tags

Return the list of tags the model has

Returns

list of strings (tags)