Go to www.clear.ml
ClearML in GitHub Community on Slack ClearML on YouTube ClearML on Twitter ClearML on Linkedin
  • Home
  • Getting Started
    • Getting Started using the Free ClearML Hosted Service
      • Getting Started
    • Your Own ClearML Server
      • ClearML Server Deployment Formats
        • AWS EC2 AMIs
        • Google Cloud Platform
        • Linux and macOS
        • Windows 10
        • Kubernetes
        • Kubernetes using Helm
        • Upgrading from Trains Server 0.15 or older to ClearML Server
      • Configuring Your Own ClearML Server
        • ClearML Server Deployment Configuration
        • ClearML Server Feature Configurations
        • Configuration procedures
      • Configuring ClearML for Your ClearML Server
        • Add ClearML to a configuration file
      • Securing Your Own ClearML Server
        • Network Security
        • User Access Security
        • Server Credentials and Secrets
  • Fundamentals
    • Tasks (Experiments)
      • Creating new Tasks
      • Data stored in Tasks
      • Tasks types
      • Task states and state transitions
    • Logging and Debug Samples
      • Automatic logging
      • Explicit reporting
      • Debug Samples
    • Artifacts
    • Storage
      • Configuring ClearML for artifact storage
      • Configuring ClearML for debug samples storage
    • Workers and Queues
  • Architecture
    • ClearML Agent
    • ClearML Python Package
      • Modules
      • Examples
    • ClearML Server
      • ClearML Agent services container
  • MLOps
    • Installing and Configuring Your ClearML Agent
      • Adding ClearML Agent to a configuration file
    • ClearML Agent Use Case Examples
      • Running workers
        • The default queue
        • User-created queues
        • Prioritizing queues
        • Docker mode
        • Specifying GPUs
        • Debugging
      • Explicit Task execution
        • Execute a Task without queue
        • Clone a Task and execute the cloned Task
        • Docker mode
      • Building Docker containers
        • Containerized Tasks
        • Base Docker image
      • Launching ClearML Agent in services mode
  • Tutorials
    • Explicit Reporting Tutorial
      • Prerequisites
      • Before you begin
      • Step 1. Setting an output destination for model checkpoints
      • Step 2. Logger class reporting methods
        • Get a logger
        • Plot scalar metrics
        • Plot other (not scalar) data
        • Log text
      • Step 3. Registering artifacts
        • Register the artifact
        • Reference the registered artifact
      • Step 4. Uploading artifacts
      • Additional information
    • Tuning Experiments Tutorial
      • Prerequisites
      • Step 1. Run the experiment
      • Step 2. Clone the experiment
      • Step 3. Tune the cloned experiment
      • Step 4. Run a worker daemon listening to a queue
      • Step 5. Enqueue the tuned experiment
      • Step 6. Compare the experiments
    • Tracking Leaderboards Tutorial
      • Step 1. Select a project
      • Step 2. Filter the experiments
      • Step 3. Hide the defaults column
      • Step 4. Show metrics or hyperparameters
      • Step 5. Enable auto refresh
      • Step 6. Save the tracking leaderboard
  • Examples
    • Automation
      • Manual Random Parameter Search
      • Task Piping
    • Distributed
      • PyTorch Distributed
        • Artifacts
        • Scalars
        • Hyperparameters
        • Log
      • Subprocess
        • Hyperparameters
        • Log
    • Frameworks
      • AutoKeras
        • Scalars
        • Hyperparameters
        • Log
        • Artifacts
      • Fastai
        • Scalars
        • Plots
        • Logs
      • Keras
        • Keras with TensorBoard - Jupyter Notebook
        • Keras with Matplotlib - Jupyter Notebook
        • Keras with TensorBoard
        • Manual Model Upload
      • Matplotlib
        • Matplotlib - Jupyter Notebook
        • Matplotlib
      • PyTorch
        • Manual Model Upload
        • PyTorch Distributed
        • PyTorch with Matplotlib
        • PyTorch MNIST
        • PyTorch with TensorBoard
        • PyTorch TensorBoardX
        • PyTorch TensorBoard Toy
      • Pytorch Notebooks
        • Audio Classification - Jupyter Notebooks
        • Audio Preprocessing - Jupyter Notebook
        • Image Hyperparameter Optimization - Jupyter Notebook
        • Image Classification - Jupyter Notebook
        • Tabular Data Downloading and Preprocessing - Jupyter Notebook
        • Text Classification - Jupyter Notebook
      • scikit-learn
        • scikit-learn with Joblib
        • scikit-learn with Matplotlib
      • TensorBoardX
        • Scalars
        • Hyperparameters
        • Log
        • Artifacts
      • TensorFlow
        • Manual Model Upload
        • TensorBoard PR Curve
        • TensorBoard Toy
        • TensorFlow MNIST
      • XGBoost
        • Plots
        • Log
        • Artifacts
    • Explicit Reporting
      • Explicit Reporting - Jupyter Notebook
        • Scalars
        • Plots
        • Media
        • Text
      • 2D Plots Reporting
        • Histograms
        • Confusion Matrices
        • 2D scatter plots
      • 3D Plots Reporting
        • Surface plot
        • 3D scatter plot
      • Artifacts Reporting
        • Dynamically tracked artifacts
        • Artifacts without tracking
      • Configuring Models
        • Configuring models
        • Label enumeration
      • HTML Reporting
        • Reporting HTML URLs
        • Reporting HTML local files
      • Hyperparameters Reporting
        • argparse command line options
        • TensorFlow Definitions
        • Parameter dictionaries
      • Manual Matplotlib Reporting
      • Images Reporting
      • Media Reporting
        • Reporting (uploading) media from a source by URL
        • Reporting (uploading) media from a local file
      • Plotly Reporting
      • Scalars Reporting
      • Tables Reporting (Pandas and CSV Files)
        • Reporting Pandas DataFrames as tables
        • Reporting CSV files as tables
      • Text Reporting
    • Optimization
      • Hyperparameter Optimization
        • Set the search strategy for optimization
        • Define a callback
        • Initialize the optimization Task
        • Setup the arguments
        • Instantiate the optimizer object
        • Running as a service
        • Optimize
    • Pipelines
      • Simple Pipeline - Serialized Data
        • The pipeline controller
        • Step 1 - Downloading the data
        • Step 2 - Processing the data
        • Step 3 - Training the network
        • Running the pipeline
      • Tabular Data Pipeline with Concurrent Steps - Jupyter Notebook
        • Pipeline controller and steps
        • Running the pipeline
    • Services
      • ClearML AWS Autoscaler Service
        • Running the ClearML AWS autoscaler
      • Cleanup Service
        • Prerequisites
        • Running the cleanup service
        • The cleanup service code
      • Jupyter Notebook Server Service
        • Running the Jupyter Notebook server service
        • Logging the Jupyter Notebook server
      • Monitoring Service Posting Slack Alerts
        • Prerequisites
        • Creating a Slack Bot
        • Running the service
        • Additional information about slack_alerts.py
    • Storage
      • Storage Examples
        • StorageManager
  • Web UI
    • Home Page
    • Projects Page
      • Experiments
        • The Experiments Table
        • Tracking Experiments and Visualizing Results
        • Reproducing experiments
        • Tuning experiments
        • Comparing Experiments
        • Sharing Experiments and Models
      • Models
        • The Models Table
        • Viewing Model Details
        • Modifying Models
      • Archiving
    • Workers and Queues Page
      • Resources utilization
      • Worker utilization
      • Queue utilization
      • Queue management
    • Profile Page
      • Setting user preferences
      • Providing Cloud Storage Access for the ClearML Web UI
      • Creating ClearML credentials
      • Switching workspaces
      • Inviting new teammates
      • Leaving a workspace
  • References
    • ClearML Python Package
      • Task
        • task.Task
          • DeleteError
          • args
          • with_traceback
          • TaskStatusEnum
          • add_requirements
          • add_tags
          • artifacts
          • cache_dir
          • clone
          • close
          • completed
          • connect
          • connect_configuration
          • connect_label_enumeration
          • create
          • create_function_task
          • current_task
          • debug_simulate_remote_task
          • delete
          • delete_parameter
          • delete_user_properties
          • dequeue
          • enqueue
          • execute_remotely
          • export_task
          • flush
          • get_all
          • get_archived
          • get_base_docker
          • get_configuration_object
          • get_initial_iteration
          • get_label_num_description
          • get_labels_enumeration
          • get_last_iteration
          • get_last_scalar_metrics
          • get_logger
          • get_model_config_dict
          • get_model_config_text
          • get_model_design
          • get_models
          • get_num_of_classes
          • get_offline_mode_folder
          • get_output_destination
          • get_output_log_web_page
          • get_parameter
          • get_parameters
          • get_parameters_as_dict
          • get_project_id
          • get_projects
          • get_registered_artifacts
          • get_reported_console_output
          • get_reported_scalars
          • get_status
          • get_task
          • get_tasks
          • get_user_properties
          • import_offline_session
          • import_task
          • init
          • input_model
          • is_current_task
          • is_main_task
          • is_offline
          • labels_stats
          • logger
          • mark_failed
          • mark_started
          • mark_stopped
          • metrics_manager
          • models
          • output_model
          • publish
          • register_artifact
          • reload
          • reset
          • running_locally
          • save_exec_model_design_file
          • set_archived
          • set_artifacts
          • set_base_docker
          • set_comment
          • set_configuration_object
          • set_credentials
          • set_initial_iteration
          • set_input_model
          • set_model_config
          • set_model_label_enumeration
          • set_name
          • set_offline
          • set_parameter
          • set_parameters
          • set_parameters_as_dict
          • set_parent
          • set_resource_monitor_iteration_timeout
          • set_task_type
          • set_user_properties
          • started
          • status
          • stopped
          • unregister_artifact
          • update_model_desc
          • update_output_model
          • update_output_model_and_upload
          • update_parameters
          • update_task
          • upload_artifact
          • wait_for_status
      • Logger
        • logger.Logger
          • capture_logging
          • current_logger
          • flush
          • get_default_upload_destination
          • get_flush_period
          • report_confusion_matrix
          • report_histogram
          • report_image
          • report_image_and_upload
          • report_line_plot
          • report_matplotlib_figure
          • report_matrix
          • report_media
          • report_plotly
          • report_scalar
          • report_scatter2d
          • report_scatter3d
          • report_surface
          • report_table
          • report_text
          • report_vector
          • set_default_upload_destination
          • set_flush_period
          • tensorboard_auto_group_scalars
          • tensorboard_single_series_per_graph
      • Dataset
        • Dataset
          • add_files
          • create
          • delete
          • file_entries_dict
          • finalize
          • get
          • get_default_storage
          • get_dependency_graph
          • get_local_copy
          • get_mutable_local_copy
          • is_dirty
          • is_final
          • list_added_files
          • list_datasets
          • list_files
          • list_modified_files
          • list_removed_files
          • remove_files
          • squash
          • sync_folder
          • upload
          • verify_dataset_hash
      • Storage
        • storage.manager.StorageManager
          • download_folder
          • get_local_copy
          • set_cache_file_limit
          • upload_file
          • upload_folder
      • Automation
        • automation.controller.PipelineController
          • add_step
          • elapsed
          • get_pipeline_dag
          • get_processed_nodes
          • get_running_nodes
          • is_running
          • start
          • stop
          • wait
        • automation.optimization.HyperParameterOptimizer
          • elapsed
          • get_active_experiments
          • get_num_active_experiments
          • get_optimizer
          • get_optimizer_top_experiments
          • get_time_limit
          • get_top_experiments
          • is_active
          • is_running
          • reached_time_limit
          • set_default_job_class
          • set_report_period
          • set_time_limit
          • start
          • stop
          • wait
        • automation.parameters.DiscreteParameterRange
          • from_dict
          • get_random_seed
          • get_value
          • set_random_seed
          • to_dict
          • to_list
        • automation.parameters.ParameterSet
          • from_dict
          • get_random_seed
          • get_value
          • set_random_seed
          • to_dict
          • to_list
        • automation.parameters.UniformIntegerParameterRange
          • from_dict
          • get_random_seed
          • get_value
          • set_random_seed
          • to_dict
          • to_list
        • automation.parameters.UniformParameterRange
          • from_dict
          • get_random_seed
          • get_value
          • set_random_seed
          • to_dict
          • to_list
        • automation.optimization.GridSearch
          • create_job
          • get_created_jobs_ids
          • get_created_jobs_tasks
          • get_objective_metric
          • get_running_jobs
          • get_top_experiments
          • helper_create_job
          • monitor_job
          • process_step
          • set_job_class
          • set_job_default_parent
          • set_job_naming_scheme
          • set_optimizer_task
          • start
          • stop
        • automation.optimization.RandomSearch
          • create_job
          • get_created_jobs_ids
          • get_created_jobs_tasks
          • get_objective_metric
          • get_running_jobs
          • get_top_experiments
          • helper_create_job
          • monitor_job
          • process_step
          • set_job_class
          • set_job_default_parent
          • set_job_naming_scheme
          • set_optimizer_task
          • start
          • stop
        • automation.optuna.optuna.OptimizerOptuna
          • create_job
          • get_created_jobs_ids
          • get_created_jobs_tasks
          • get_objective_metric
          • get_running_jobs
          • get_top_experiments
          • helper_create_job
          • monitor_job
          • process_step
          • set_job_class
          • set_job_default_parent
          • set_job_naming_scheme
          • set_optimizer_task
          • start
          • stop
        • automation.hpbandster.bandster.OptimizerBOHB
          • create_job
          • get_created_jobs_ids
          • get_created_jobs_tasks
          • get_objective_metric
          • get_random_seed
          • get_running_jobs
          • get_top_experiments
          • helper_create_job
          • monitor_job
          • process_step
          • set_job_class
          • set_job_default_parent
          • set_job_naming_scheme
          • set_optimization_args
          • set_optimizer_task
          • set_random_seed
          • start
          • stop
      • Model
        • model.InputModel
          • comment
          • config_dict
          • config_text
          • connect
          • empty
          • get_local_copy
          • get_weights
          • get_weights_package
          • id
          • import_model
          • labels
          • load_model
          • name
          • publish
          • system_tags
          • tags
          • task
          • url
        • model.Model
          • comment
          • config_dict
          • config_text
          • get_local_copy
          • get_weights
          • get_weights_package
          • id
          • labels
          • name
          • publish
          • system_tags
          • tags
          • task
          • url
        • model.OutputModel
          • comment
          • config_dict
          • config_text
          • connect
          • get_weights
          • get_weights_package
          • id
          • labels
          • name
          • publish
          • published
          • set_upload_destination
          • system_tags
          • tags
          • task
          • update_design
          • update_labels
          • update_weights
          • update_weights_package
          • url
          • wait_for_uploads
      • ClearML Python Package Extras
        • Step 1. Install ClearML extras for storage
        • Step 2. Initializing a new ClearML configuration file
        • Step 3. Add storage credentials to your ClearML configuration file
        • AWS S3
        • Azure Storage
        • Google Cloud Storage
    • ClearML Agent
      • build
        • Syntax
        • Arguments
      • config
        • Syntax
      • daemon
        • Syntax
        • Arguments
      • execute
        • Syntax
        • Arguments
      • list
        • Syntax
    • ClearML Server API (self-hosted)
      • auth.login
      • auth.logout
      • auth.get_token_for_user
      • auth.validate_token
      • auth.create_user
      • auth.create_credentials
      • auth.get_credentials
      • auth.revoke_credentials
      • auth.edit_user
      • auth.fixed_users_mode
      • debug.ping
      • events.add
      • events.add_batch
      • events.delete_for_task
      • events.debug_images
      • events.get_debug_image_sample
      • events.next_debug_image_sample
      • events.get_task_metrics
      • events.get_task_log
      • events.get_task_events
      • events.download_task_log
      • events.get_task_plots
      • events.get_multi_task_plots
      • events.get_vector_metrics_and_variants
      • events.vector_metrics_iter_histogram
      • events.scalar_metrics_iter_histogram
      • events.multi_task_scalar_metrics_iter_histogram
      • events.get_task_latest_scalar_values
      • events.get_scalar_metrics_and_variants
      • events.get_scalar_metric_data
      • login.supported_modes
      • models.get_by_id
      • models.get_by_task_id
      • models.get_by_id_ex
      • models.get_all_ex
      • models.get_all
      • models.get_frameworks
      • models.update_for_task
      • models.create
      • models.edit
      • models.update
      • models.set_ready
      • models.delete
      • models.make_public
      • models.make_private
      • models.move
      • organization.get_tags
      • organization.get_user_companies
      • projects.create
      • projects.get_by_id
      • projects.get_all
      • projects.get_all_ex
      • projects.update
      • projects.delete
      • projects.get_unique_metric_variants
      • projects.get_hyperparam_values
      • projects.get_hyper_parameters
      • projects.get_task_tags
      • projects.get_model_tags
      • projects.make_public
      • projects.make_private
      • projects.get_task_parents
      • queues.get_by_id
      • queues.get_all_ex
      • queues.get_all
      • queues.get_default
      • queues.create
      • queues.update
      • queues.delete
      • queues.add_task
      • queues.get_next_task
      • queues.remove_task
      • queues.move_task_forward
      • queues.move_task_backward
      • queues.move_task_to_front
      • queues.move_task_to_back
      • queues.get_queue_metrics
      • server.get_stats
      • server.config
      • server.info
      • server.endpoints
      • server.report_stats_option
      • tasks.get_by_id
      • tasks.get_by_id_ex
      • tasks.get_all_ex
      • tasks.get_all
      • tasks.get_types
      • tasks.clone
      • tasks.create
      • tasks.validate
      • tasks.update
      • tasks.update_batch
      • tasks.edit
      • tasks.reset
      • tasks.delete
      • tasks.archive
      • tasks.started
      • tasks.stop
      • tasks.stopped
      • tasks.failed
      • tasks.close
      • tasks.publish
      • tasks.enqueue
      • tasks.dequeue
      • tasks.set_requirements
      • tasks.completed
      • tasks.ping
      • tasks.add_or_update_artifacts
      • tasks.delete_artifacts
      • tasks.make_public
      • tasks.make_private
      • tasks.get_hyper_params
      • tasks.edit_hyper_params
      • tasks.delete_hyper_params
      • tasks.get_configurations
      • tasks.get_configuration_names
      • tasks.edit_configuration
      • tasks.delete_configuration
      • tasks.move
      • users.get_by_id
      • users.get_current_user
      • users.get_all_ex
      • users.get_all
      • users.delete
      • users.create
      • users.update
      • users.get_preferences
      • users.set_preferences
      • workers.get_all
      • workers.register
      • workers.unregister
      • workers.status_report
      • workers.get_metric_keys
      • workers.get_stats
      • workers.get_activity_report
    • ClearML Configuration
      • Editing your configuration file
      • agent section
        • agent.default_docker
        • agent.package_manager
        • agent.pip_download_cache
        • agent.vcs_cache
        • agent.venv_update
      • api section
        • api.credentials
      • sdk section
        • sdk.aws
        • sdk.azure
        • sdk.development
        • sdk.google.storage
        • sdk.log
        • sdk.metrics
        • sdk.network
        • sdk.storage
  • Integrations
    • AutoKeras
      • Adding ClearML to code
    • Keras Tuner
      • ClearMLTunerLogger
      • Scalars
      • Summary of hyperparameter optimization
      • Artifacts
      • Configuration objects
        • Hyperparameters
        • Configuration
    • PyTorch Ignite
      • Ignite ClearMLLogger
        • ClearMLLogger parameters
      • Visualizing experiment results
        • Scalars
        • Model snapshots
      • Logging
        • Ignite engine output and / or metrics
        • Optimizer parameters
        • Model weights
      • Model snapshots
      • MNIST example
    • Git for Jupyter Notebook
      • Installation
      • Using the plugin
      • Screenshots
      • Acknowledgements
    • Integration for PyCharm
      • Installation
      • Optional: ClearML configuration parameters
  • FAQ
    • General Information
    • Models
    • Experiments
    • Graphs and Logs
    • GIT and Storage
    • Jupyter
    • Remote Debugging (ClearML PyCharm Plugin)
    • scikit-learn
    • ClearML Configuration
    • ClearML Hosted Service
    • ClearML Server Deployment
    • ClearML Server Configuration
    • ClearML Server Troubleshooting
  • Community Resources
    • Join the ClearML Conversation
    • Allegro AI resources
    • Guidelines for Contributing
    • Reporting Issues
    • Suggesting New Features and Enhancements
    • Pull Requests
  • Release Notes
    • Version 0.17
      • Version 0.17.4
        • ClearML
      • Version 0.17.3
        • ClearML
      • Version 0.17.2
        • ClearML
      • Version 0.17.1
        • ClearML
      • Version 0.17.0
        • ClearML
        • Open Source ClearML Server
        • ClearML Hosted Service only
        • ClearML Agent
    • Version 0.16
      • Version 0.16.4
        • Trains
      • Version 0.16.3
        • Trains
      • Version 0.16.2
        • Trains
        • Trains-Agent
      • Version 0.16.1
        • Trains
        • Trains Server
        • Trains Agent
      • Version 0.16.0
        • Trains
        • Trains Server
        • Trains Agent
    • Version 0.15
      • Version 0.15.1
        • Trains
        • Trains Server
        • Trains Agent
      • Version 0.15.0
        • Trains
        • Trains Server
        • Trains Agent
    • Version 0.14
      • Version 0.14.3
        • Trains
      • Version 0.14.2
        • Trains
        • Trains Server
      • Version 0.14.1
        • Trains
        • Trains Server
        • Trains Agent
      • Version 0.14.0
        • Trains
        • Trains Server
        • Trains Agent
    • Version 0.13
      • Version 0.13.3
        • Trains
        • Trains Agent
      • Version 0.13.2
        • Trains
      • Version 0.13.1
        • Trains
      • Version 0.13.0
        • Trains
        • Trains Server
        • Trains Agent
    • Version 0.12
      • Version 0.12.2
        • Trains
        • Trains Agent
      • Version 0.12.1
        • Trains
        • Trains Server
        • Trains Agent
      • Version 0.12.0
        • Trains
        • Trains Server
        • Trains Agent
    • Version 0.11
      • Version 0.11.3
        • Trains
      • Version 0.11.2
        • Trains
      • Version 0.11.1
        • Trains
      • Version 0.11.0
        • Trains
        • Trains Server
    • Version 0.10
      • Version 0.10.7
        • Trains
      • Version 0.10.6
        • Trains
      • Version 0.10.5
        • Trains
      • Version 0.10.4
        • Trains
      • Version 0.10.3
        • Trains
      • Version 0.10.2
        • Trains
      • Version 0.10.1
        • Trains
        • Trains Server
      • Version 0.10.0
        • Trains
        • Trains Server
    • Version 0.9
      • Version 0.9.3
        • Trains
      • Version 0.9.2
        • Trains
      • Version 0.9.1
        • Trains
ClearML Documentation
  • »
  • Home »
  • Storage

Storage¶

  • Storage Examples
Previous
Monitoring Service Posting Slack Alerts
Next
Storage Examples
© 2020 • Allegro AI. All Rights Reserved clearml@allegro.ai Built with Sphinx using a theme provided by Read the Docs.