Storage Examples
Trains is now ClearML
This documentation applies to the legacy Trains versions. For the latest documentation, see ClearML.
This page describes storage examples using the StorageManager class.
The storage examples include:
- Downloading a file - Get an object from storage.
- Uploading a file - Upload an object.
- Setting cache limits - Set the maximum number of objects.
StorageManager
Downloading a file
To download an ZIP file from storage to the global
cache context, call the StorageManager.get_local_copy method, and specifying the destination location as the remote_url
argument.
# create a StorageManager instance
manager = StorageManager()
manager.get_local_copy(remote_url="s3://MyBucket/MyFolder/file.zip")
To download a file to specific context in cache,
and specify the cache_context
argument as the name of the context.
manager.get_local_copy(remote_url="s3://MyBucket/MyFolder/file.ext", cache_context="test")
To download a non-compressed file, set the extract_archive
argument as False
.
manager.get_local_copy(remote_url="s3://MyBucket/MyFolder/file.ext", extract_archive=False
Uploading a file
To upload a file to storage, call the StorageManager.upload_file method. Specify the full path of the local file as the local_file
parameter argument, and the remote URL as the remote_url
parameter argument.
manager.upload_file(local_file="/mnt/data/also_file.ext", remote_url="s3://MyBucket/MyFolder")
Setting cache limits
To set a limit on the number of files cached, call the StorageManager.set_cache_file_limit method and specify the cache_file_limit
argument as the maximum number of files. This does not limit the cache size, only the number of files.
new_cache_limit = manager.set_cache_file_limit(cache_file_limit=100)