Storage Examples

Trains is now ClearML

This documentation applies to the legacy Trains versions. For the latest documentation, see ClearML.

This page describes storage examples using the StorageManager class.

The storage examples include:

• Downloading a file - Get an object from storage.
• Uploading a file - Upload an object.
• Setting cache limits - Set the maximum number of objects.

StorageManager

Downloading a file

To download an ZIP file from storage to the global cache context, call the StorageManager.get_local_copy method, and specifying the destination location as the remote_url argument.

# create a StorageManager instance
manager = StorageManager()

manager.get_local_copy(remote_url="s3://MyBucket/MyFolder/file.zip")

To download a file to specific context in cache, and specify the cache_context argument as the name of the context.

manager.get_local_copy(remote_url="s3://MyBucket/MyFolder/file.ext", cache_context="test")

To download a non-compressed file, set the extract_archive argument as False.

manager.get_local_copy(remote_url="s3://MyBucket/MyFolder/file.ext", extract_archive=False

Uploading a file

To upload a file to storage, call the StorageManager.upload_file method. Specify the full path of the local file as the local_file parameter argument, and the remote URL as the remote_url parameter argument.

manager.upload_file(local_file="/mnt/data/also_file.ext", remote_url="s3://MyBucket/MyFolder")

Setting cache limits

To set a limit on the number of files cached, call the StorageManager.set_cache_file_limit method and specify the cache_file_limit argument as the maximum number of files. This does not limit the cache size, only the number of files.

new_cache_limit = manager.set_cache_file_limit(cache_file_limit=100)