trains.model.
InputModel
¶Load an existing model in the system, search by model id. The Model will be read-only and can be used to pre initialize a network We can connect the model to a task as input model, then when running remotely override it with the UI.
model_id (str) – The Trains Id (system UUID) of the input model whose metadata the Trains Server (backend) stores.
import_model
(weights_url, config_text=None, config_dict=None, label_enumeration=None, name=None, tags=None, comment=None, is_package=False, create_as_published=False, framework=None)¶Create an InputModel object from a pre-trained model by specifying the URL of an initial weight files.
Optionally, input a configuration, label enumeration, name for the model, tags describing the model,
comment as a description of the model, indicate whether the model is a package, specify the model’s
framework, and indicate whether to immediately set the model’s status to Published
.
The model is read-only.
The Trains Server (backend) may already store the model’s URL. If the input model’s URL is not stored, meaning the model is new, then it is imported and Trains stores its metadata. If the URL is already stored, the import process stops, Trains issues a warning message, and Trains reuses the model.
In your Python experiment script, after importing the model, you can connect it to the main execution
Task as an input model using InputModel.connect
or Task.connect
. That initializes the
network.
Note
Using the Trains Web-App (user interface), you can reuse imported models and switch models in experiments.
weights_url (str) –
A valid URL for the initial weights file. If the Trains Web-App (backend) already stores the metadata of a model with the same URL, that existing model is returned and Trains ignores all other parameters.
For example:
https://domain.com/file.bin
s3://bucket/file.bin
file:///home/user/file.bin
config_text (unconstrained text string) – The configuration as a string. This is usually the content of a configuration
dictionary file. Specify config_text
or config_dict
, but not both.
config_dict (dict) – The configuration as a dictionary. Specify config_text
or config_dict
,
but not both.
label_enumeration (dict) –
Optional label enumeration dictionary of string (label) to integer (value) pairs.
For example:
{
'background': 0,
'person': 1
}
name (str) – The name of the newly imported model. (Optional)
tags (list(str)) – The list of tags which describe the model. (Optional)
comment (str) – A comment / description for the model. (Optional)
is_package (bool) –
Is the imported weights file is a package (Optional)
True
- Is a package. Add a package tag to the model.
False
- Is not a package. Do not add a package tag. (Default)
create_as_published (bool) –
Set the model’s status to Published (Optional)
True
- Set the status to Published.
False
- Do not set the status to Published. The status will be Draft. (Default)
framework (str or Framework object) – The framework of the model. (Optional)
The imported model or existing model (see above).
load_model
(weights_url, load_archived=False)¶Load an already registered model based on a pre-existing model file (link must be valid). If the url to the
weights file already exists, the returned object is a Model representing the loaded Model. If no registered
model with the specified url is found, None
is returned.
weights_url –
The valid url for the weights file (string).
Examples:
"https://domain.com/file.bin" or "s3://bucket/file.bin" or "file:///home/user/file.bin".
Note
If a model with the exact same URL exists, it will be used, and all other arguments will be ignored.
load_archived (bool) –
Load archived models
True
- Load the registered Model, if it is archived.
False
- Ignore archive models.
The InputModel object, or None if no model could be found.
empty
(config_text=None, config_dict=None, label_enumeration=None)¶Create an empty model object. Later, you can assign a model to the empty model object.
config_text (unconstrained text string) – The model configuration as a string. This is usually the content of a configuration
dictionary file. Specify config_text
or config_dict
, but not both.
config_dict (dict) – The model configuration as a dictionary. Specify config_text
or config_dict
,
but not both.
label_enumeration (dict) –
The label enumeration dictionary of string (label) to integer (value) pairs. (Optional)
For example:
{
'background': 0,
'person': 1
}
An empty model object.
id
¶The Id (system UUID) of the model.
The model ID.
connect
(task)¶Connect the current model to a Task object, if the model is preexisting. Preexisting models include:
Imported models (InputModel objects created using the Logger.import_model
method).
Models whose metadata is already in the Trains platform, meaning the InputModel object is instantiated
from the InputModel
class specifying the the model’s Trains Id as an argument.
Models whose origin is not Trains that are used to create an InputModel object. For example, models created using TensorFlow models.
When the experiment is executed remotely in a worker, the input model already specified in the experiment is used.
Note
The Trains Web-App allows you to switch one input model for another and then enqueue the experiment to execute in a worker.
task (object) – A Task object.
comment
¶The comment for the model. Also, use for a model description.
The model comment / description.
config_dict
¶The configuration as a dictionary, parsed from the design text. This usually represents the model configuration. For example, prototxt, an ini file, or Python code to evaluate.
The configuration.
config_text
¶The configuration as a string. For example, prototxt, an ini file, or Python code to evaluate.
The configuration.
get_local_copy
(extract_archive=True, raise_on_error=False)¶Retrieve a valid link to the model file(s). If the model URL is a file system link, it will be returned directly. If the model URL is points to a remote location (http/s3/gs etc.), it will download the file(s) and return the temporary location of the downloaded model.
extract_archive (bool) – If True and the model is of type ‘packaged’ (e.g. TensorFlow compressed folder) The returned path will be a temporary folder containing the archive content
raise_on_error (bool) – If True and the artifact could not be downloaded, raise ValueError, otherwise return None on failure and output log warning.
A local path to the model (or a downloaded copy of it).
get_weights
(raise_on_error=False)¶Download the base model and return the locally stored filename.
raise_on_error (bool) – If True and the artifact could not be downloaded, raise ValueError, otherwise return None on failure and output log warning.
The locally stored file.
get_weights_package
(return_path=False, raise_on_error=False)¶Download the base model package into a temporary directory (extract the files), or return a list of the locally stored filenames.
return_path (bool) –
Return the model weights or a list of filenames (Optional)
True
- Download the model weights into a temporary directory, and return the temporary directory path.
False
- Return a list of the locally stored filenames. (Default)
raise_on_error (bool) – If True and the artifact could not be downloaded, raise ValueError, otherwise return None on failure and output log warning.
The model weights, or a list of the locally stored filenames.
labels
¶The label enumeration of string (label) to integer (value) pairs.
A dictionary containing labels enumeration, where the keys are labels and the values as integers.
name
¶The name of the model.
The model name.
publish
()¶Set the model to the status published
and for public use. If the model’s status is already published
,
then this method is a no-op.
A list of system tags describing the model.
The list of tags.
A list of tags describing the model.
The list of tags.
task
¶Return the creating task object
The Task object.
url
¶Return the url of the model file (or archived files)
The model file URL.
trains.model.
OutputModel
¶Create an output model for a Task (experiment) to store the training results.
The OutputModel object is always connected to a Task object, because it is instantiated with a Task object as an argument. It is, therefore, automatically registered as the Task’s (experiment’s) output model.
The OutputModel object is read-write.
A common use case is to reuse the OutputModel object, and override the weights after storing a model snapshot. Another use case is to create multiple OutputModel objects for a Task (experiment), and after a new high score is found, store a model snapshot.
If the model configuration and / or the model’s label enumeration
are None
, then the output model is initialized with the values from the Task object’s input model.
Note
When executing a Task (experiment) remotely in a worker, you can modify the model configuration and / or model’s label enumeration using the Trains Web-App.
Create a new model and immediately connect it to a task.
We do not allow for Model creation without a task, so we always keep track on how we created the models In remote execution, Model parameters can be overridden by the Task (such as model configuration & label enumerator)
task (Task) – The Task object with which the OutputModel object is associated.
config_text (unconstrained text string) – The configuration as a string. This is usually the content of a configuration
dictionary file. Specify config_text
or config_dict
, but not both.
config_dict (dict) – The configuration as a dictionary.
Specify config_dict
or config_text
, but not both.
label_enumeration (dict) –
The label enumeration dictionary of string (label) to integer (value) pairs. (Optional)
For example:
{
'background': 0,
'person': 1
}
name (str) – The name for the newly created model. (Optional)
tags (list(str)) – A list of strings which are tags for the model. (Optional)
comment (str) – A comment / description for the model. (Optional)
framework (str or Framework object) – The framework of the model or a Framework object. (Optional)
base_model_id – optional, model id to be reused
published
¶Get the published state of this model.
config_text
¶Get the configuration as a string. For example, prototxt, an ini file, or Python code to evaluate.
The configuration.
config_dict
¶Get the configuration as a dictionary parsed from the config_text
text. This usually represents the model
configuration. For example, from prototxt to ini file or python code to evaluate.
The configuration.
labels
¶Get the label enumeration as a dictionary of string (label) to integer (value) pairs.
For example:
{
'background': 0,
'person': 1
}
The label enumeration.
connect
(task)¶Connect the current model to a Task object, if the model is a preexisting model. Preexisting models include:
Imported models.
Models whose metadata the Trains Server (backend) is already storing.
Models from another source, such as frameworks like TensorFlow.
task (object) – A Task object.
set_upload_destination
(uri)¶Set the URI of the storage destination for uploaded model weight files. Supported storage destinations include S3, Google Cloud Storage), and file locations.
Using this method, files uploads are separate and then a link to each is stored in the model object.
Note
For storage requiring credentials, the credentials are stored in the Trains configuration file,
~/trains.conf
.
uri (str) –
The URI of the upload storage destination.
For example:
s3://bucket/directory/
file:///tmp/debug/
The status of whether the storage destination schema is supported.
True
- The storage destination scheme is supported.
False
- The storage destination scheme is not supported.
update_weights
(weights_filename=None, upload_uri=None, target_filename=None, auto_delete_file=True, register_uri=None, iteration=None, update_comment=True, is_package=False)¶Update the model weights from a locally stored model filename.
Note
Uploading the model is a background process. A call to this method returns immediately.
weights_filename (str) – The name of the locally stored weights file to upload.
Specify weights_filename
or register_uri
, but not both.
upload_uri (str) – The URI of the storage destination for model weights upload. The default value is the previously used URI. (Optional)
target_filename (str) – The newly created filename in the storage destination location. The default value
is the weights_filename
value. (Optional)
auto_delete_file (bool) –
Delete the temporary file after uploading (Optional)
True
- Delete (Default)
False
- Do not delete
register_uri (str) – The URI of an already uploaded weights file. The URI must be valid. Specify
register_uri
or weights_filename
, but not both.
iteration (int) – The iteration number.
update_comment (bool) –
Update the model comment with the local weights file name (to maintain provenance) (Optional)
True
- Update model comment (Default)
False
- Do not update
is_package (bool) – Mark the weights file as compressed package, usually a zip file.
The uploaded URI.
update_weights_package
(weights_filenames=None, weights_path=None, upload_uri=None, target_filename=None, auto_delete_file=True, iteration=None)¶Update the model weights from locally stored model files, or from directory containing multiple files.
Note
Uploading the model weights is a background process. A call to this method returns immediately.
weights_filenames (list(str)) – The file names of the locally stored model files. Specify weights_filenames
,
or weights_path
, but not both.
weights_path (str) – The directory path to a package. All the files in the directory will be uploaded.
Specify weights_path
or weights_filenames
, but not both.
upload_uri (str) – The URI of the storage destination for the model weights upload. The default is the previously used URI. (Optional)
target_filename (str) – The newly created filename in the storage destination URI location. The default
is the value specified in the weights_filename
parameter. (Optional)
auto_delete_file (bool) –
Delete temporary file after uploading (Optional)
True
- Delete (Default)
False
- Do not delete
iteration (int) – The iteration number.
The uploaded URI for the weights package.
update_design
(config_text=None, config_dict=None)¶Update the model configuration. Store a blob of text for custom usage.
Note
This method’s behavior is lazy. The design update is only forced when the weights are updated.
config_text (unconstrained text string) – The configuration as a string. This is usually the content of a configuration
dictionary file. Specify config_text
or config_dict
, but not both.
config_dict (dict) – The configuration as a dictionary. Specify config_text
or config_dict
,
but not both.
True, update successful. False, update not successful.
comment
¶The comment for the model. Also, use for a model description.
The model comment / description.
get_weights
(raise_on_error=False)¶Download the base model and return the locally stored filename.
raise_on_error (bool) – If True and the artifact could not be downloaded, raise ValueError, otherwise return None on failure and output log warning.
The locally stored file.
get_weights_package
(return_path=False, raise_on_error=False)¶Download the base model package into a temporary directory (extract the files), or return a list of the locally stored filenames.
return_path (bool) –
Return the model weights or a list of filenames (Optional)
True
- Download the model weights into a temporary directory, and return the temporary directory path.
False
- Return a list of the locally stored filenames. (Default)
raise_on_error (bool) – If True and the artifact could not be downloaded, raise ValueError, otherwise return None on failure and output log warning.
The model weights, or a list of the locally stored filenames.
id
¶The Id (system UUID) of the model.
The model ID.
name
¶The name of the model.
The model name.
publish
()¶Set the model to the status published
and for public use. If the model’s status is already published
,
then this method is a no-op.
A list of system tags describing the model.
The list of tags.
A list of tags describing the model.
The list of tags.
task
¶Return the creating task object
The Task object.
update_labels
(labels)¶Update the label enumeration.
labels (dict) –
The label enumeration dictionary of string (label) to integer (value) pairs.
For example:
{
'background': 0,
'person': 1
}
url
¶Return the url of the model file (or archived files)
The model file URL.
wait_for_uploads
(timeout=None, max_num_uploads=None)¶Wait for any pending or in-progress model uploads to complete. If no uploads are pending or in-progress,
then the wait_for_uploads
returns immediately.
timeout (float) – The timeout interval to wait for uploads (seconds). (Optional).
max_num_uploads (int) – The maximum number of uploads to wait for. (Optional).
trains.model.
Model
¶Represent an existing model in the system, search by model id. The Model will be read-only and can be used to pre initialize a network
Load model based on id, returned object is read-only and can be connected to a task
Notice, we can override the input model when running remotely
model_id – id (string)
get_local_copy
(extract_archive=True, raise_on_error=False)¶Retrieve a valid link to the model file(s). If the model URL is a file system link, it will be returned directly. If the model URL is points to a remote location (http/s3/gs etc.), it will download the file(s) and return the temporary location of the downloaded model.
extract_archive (bool) – If True and the model is of type ‘packaged’ (e.g. TensorFlow compressed folder) The returned path will be a temporary folder containing the archive content
raise_on_error (bool) – If True and the artifact could not be downloaded, raise ValueError, otherwise return None on failure and output log warning.
A local path to the model (or a downloaded copy of it).
comment
¶The comment for the model. Also, use for a model description.
The model comment / description.
config_dict
¶The configuration as a dictionary, parsed from the design text. This usually represents the model configuration. For example, prototxt, an ini file, or Python code to evaluate.
The configuration.
config_text
¶The configuration as a string. For example, prototxt, an ini file, or Python code to evaluate.
The configuration.
get_weights
(raise_on_error=False)¶Download the base model and return the locally stored filename.
raise_on_error (bool) – If True and the artifact could not be downloaded, raise ValueError, otherwise return None on failure and output log warning.
The locally stored file.
get_weights_package
(return_path=False, raise_on_error=False)¶Download the base model package into a temporary directory (extract the files), or return a list of the locally stored filenames.
return_path (bool) –
Return the model weights or a list of filenames (Optional)
True
- Download the model weights into a temporary directory, and return the temporary directory path.
False
- Return a list of the locally stored filenames. (Default)
raise_on_error (bool) – If True and the artifact could not be downloaded, raise ValueError, otherwise return None on failure and output log warning.
The model weights, or a list of the locally stored filenames.
id
¶The Id (system UUID) of the model.
The model ID.
labels
¶The label enumeration of string (label) to integer (value) pairs.
A dictionary containing labels enumeration, where the keys are labels and the values as integers.
name
¶The name of the model.
The model name.
publish
()¶Set the model to the status published
and for public use. If the model’s status is already published
,
then this method is a no-op.
A list of system tags describing the model.
The list of tags.
A list of tags describing the model.
The list of tags.
task
¶Return the creating task object
The Task object.
url
¶Return the url of the model file (or archived files)
The model file URL.