Storage Examples¶
This page describes storage examples using the StorageManager class. The storage examples include:
Downloading a file - Get an object from storage.
Uploading a file - Upload an object.
Setting cache limits - Set the maximum number of objects.
Note
StorageManager
supports http(s), S3, Google Cloud Storage, Azure, and file system folders.
StorageManager¶
Downloading a file¶
To download an ZIP file from storage to the global
cache context, call the StorageManager.get_local_copy method, and specifying the destination location as the remote_url
argument.
# create a StorageManager instance
manager = StorageManager()
manager.get_local_copy(remote_url="s3://MyBucket/MyFolder/file.zip")
To download a file to specific context in cache,
and specify the cache_context
argument as the name of the context.
manager.get_local_copy(remote_url="s3://MyBucket/MyFolder/file.ext", cache_context="test")
To download a non-compressed file, set the extract_archive
argument as False
.
manager.get_local_copy(remote_url="s3://MyBucket/MyFolder/file.ext", extract_archive=False
Uploading a file¶
To upload a file to storage, call the StorageManager.upload_file method. Specify the full path of the local file as the local_file
parameter argument, and the remote URL as the remote_url
parameter argument.
manager.upload_file(local_file="/mnt/data/also_file.ext", remote_url="s3://MyBucket/MyFolder")
Setting cache limits¶
To set a limit on the number of files cached, call the StorageManager.set_cache_file_limit method and specify the cache_file_limit
argument as the maximum number of files. This does not limit the cache size, only the number of files.
new_cache_limit = manager.set_cache_file_limit(cache_file_limit=100)