storage.manager.StorageManager¶

class trains.storage.manager.StorageManager¶

StorageManager is helper interface for downloading and uploading files to supported remote storage. Support remote servers: http(s)/S3/GS/Azure/File-System-Folder Cache is enabled by default for all downloaded remote urls/files.

classmethod get_local_copy(remote_url, cache_context=None, extract_archive=True, name=None)¶

Get a local copy of the remote file. If the remote URL is a direct file access, the returned link is the same, otherwise a link to a local copy of the url file is returned. Caching is enabled by default, cache limited by number of stored files per cache context. Oldest accessed files are deleted when cache is full.

Parameters

remote_url (str) – remote url link (string)
cache_context (str) – Optional caching context identifier (string), default context ‘global’
extract_archive (bool) – if True returned path will be a cached folder containing the archive’s content, currently only zip files are supported.
name – name of artifact.

Returns

Full path to local copy of the requested url. Return None on Error.

classmethod set_cache_file_limit(cache_file_limit, cache_context=None)¶

Set the cache context file limit. File limit is the maximum number of files the specific cache context holds. Notice, there is no limit on the size of these files, only the total number of cached files.

Parameters

cache_file_limit (int) – New maximum number of cached files
cache_context (str) – Optional cache context identifier, default global context

Returns

The new cache context file limit.

classmethod upload_file(local_file, remote_url, wait_for_upload=True)¶

Upload a local file to a remote location. remote url is the finale destination of the uploaded file.

Examples:

upload_file('/tmp/artifact.yaml', 'http://localhost:8081/manual_artifacts/my_artifact.yaml')
 upload_file('/tmp/artifact.yaml', 's3://a_bucket/artifacts/my_artifact.yaml')
 upload_file('/tmp/artifact.yaml', '/mnt/share/folder/artifacts/my_artifact.yaml')
 

Parameters

local_file (str) – Full path of a local file to be uploaded
remote_url (str) – Full path or remote url to upload to (including file name)
wait_for_upload (bool) – If False, return immediately and upload in the background. Default True.

Returns

Newly uploaded remote URL.

Table Of Contents

storage.manager.StorageManager¶