ClearML Python Package Extras¶

This page describes how to install and configure ClearML Python Package Extras. Currently, the ClearML Python Package Extras include extras for experiment storage of model checkpoints (snapshots) and other artifacts. The supported Cloud storage types include:

AWS S3
Azure Storage
Google Cloud Storage

In addition to installing a ClearML extra for storage, you must add your storage bucket names, credentials, and related information to the ClearML configuration file.

Step 1. Install ClearML extras for storage¶

To install a ClearML extra for storage, do the following:

For your Cloud storage, execute one of the following commands:

AWS S3
pip install clearml[s3]
Google Cloud Storage
pip install clearml[gs]
Azure Storage
pip install clearml[azure]

Step 2. Initializing a new ClearML configuration file¶

Use the clearml-init setup wizard to initialize a new ClearML configuration file.

If you already have a configuration file, the setup wizard will notify you. You can then skip to the [Step 3. Add storage credentials to your ClearML configuration file](#credentials), where you add ClearML settings to an existing ClearML configuration file.

Note

Why would a ClearML configuration file already exist? If you previously configured ClearML for ClearML Server, or installed and configure ClearML Agent, you may have an existing configuration file. This is because ClearML and ClearML Agent use the same configuration file, but not all the same configuration options.

To initialize a new ClearML configuration file, do the following:

Open a terminal session.
In your terminal session, run the setup wizard.
clearml-init
If the setup wizard’s response indicates you already have a configuration file, then go to [Step 3. Add storage credentials to your ClearML configuration file](#credentials) and add your storage credentials. The wizard does not edit or overwrite existing configuration files.
Note

You can create additional ClearML configuration files using the –file option of clearml-init.

For example, create a configuration file to use inside Docker containers when executing a Task:
clearml-init --file MyOtherClearML.conf
and then specify it using the CLEARML_CONFIG_FILE environment variable inside the container:
CLEARML_CONFIG_FILE = MyOtherClearML.conf
For more information about running experiments inside Docker containers, see [ClearML Agent Use Case Examples](../../use_cases/clearml_agent_use_case_examples/) and [ClearML Agent Reference](../../references/clearml_agent_ref/) pages.
The setup begins.
CLEARML SDK setup process Please create new clearml credentials through the profile page in your clearml web app (e.g., http://localhost:8080/profile) In the profile page, press "Create new credentials", then press "Copy to clipboard". Paste copied configuration here:
At the command prompt Paste copied configuration here:, copy and paste the ClearML credentials you create as follows:
1. Open the ClearML Web UI in your browser (if you are using are demo ClearML Server, go to [https://demoapp.clearml.allegro.ai/dashboard](https://demoapp.clearml.allegro.ai/dashboard)).
2. Click the PROFILE page.
3. Click Create new credentials.
4. Click Copy to clipboard.
5. In your terminal session, paste your credentials and press Enter.

Enter the API server (press Enter to accept the default value which is based on your ClearML credentials):

Detected credentials key="********************" secret="*******"

API Host configured to: [https://demoapi.clearml.allegro.ai]

Enter your ClearML Server web server (press Enter to accept the default value which is based on your previous response):

demoapi.clearml.allegro.ai is the api server, we need the web server. Replacing 'demoapi.' with 'demoapp.'
Web Application Host configured to: [https://demoapp.clearml.allegro.ai]

Enter your ClearML file server (press Enter to accept the default value which is based on your previous response):
File Store Host configured to: [https://demofiles.clearml.allegro.ai]

Your ClearML configuration file is verified and created.

CLEARML Hosts configuration:
Web App: https://demoapp.clearml.allegro.ai
API: https://demoapi.clearml.allegro.ai
File Store: https://demofiles.clearml.allegro.ai

Verifying credentials ...
Credentials verified!

New configuration stored in /home/<username>/clearml.conf
CLEARML setup completed successfully.

The configuration file’s location depends upon your operating system:

Linux - ~/clearml.conf
Mac - $HOME/clearml.conf
Windows - User<username>clearml.conf

ref: credentials <credentials>

Step 3. Add storage credentials to your ClearML configuration file¶

AWS S3¶

For AWS S3, you can use default credentials for all buckets, or specify credentials per individual buckets.

Note

See the [AWS documentation](https://docs.aws.amazon.com/AmazonS3/latest/dev/BucketRestrictions.html) for restrictions and limitations on bucket naming.

S3 default credentials - In your configuration file aws.s3 section, insert your key, secret, and region default credentials.

aws {
    s3 {
        # S3 credentials, used for read/write access by various SDK elements

        # default, used for any bucket not specified below
        key: "my-access-key"
        secret: "my-secret-key"
        region: "my-region"
        credentials: [
            # specifies key/secret credentials to use when handling s3 urls (read or write)
            # {
            #     bucket: "my-bucket-name"
            #     key: "my-access-key"
            #     secret: "my-secret-key"
            # },
            # {
            #     # This will apply to all buckets in this host (unless key/value is specifically provided for a given bucket)
            #     host: "my-minio-host:9000"
            #     key: "12345678"
            #     secret: "12345678"
            #     multipart: false
            #     secure: false
            # }
        ]
    }
    boto3 {
        pool_connections: 512
        max_multipart_concurrency: 16
    }
}

S3 credentials per individual bucket - In the credentials section, uncomment the lines for a set of bucket credentials and insert your bucket, key, and secret.

aws {
    s3 {
        # S3 credentials, used for read/write access by various SDK elements

        # default, used for any bucket not specified below
        key: ""
        secret: ""
        region: ""

        credentials: [
            # specifies key/secret credentials to use when handling s3 urls (read or write)
            {
                bucket: "my-bucket-name"
                key: "my-access-key"
                secret: "my-secret-key"
            },
            # {
            #     # This will apply to all buckets in this host (unless key/value is specifically provided for a given bucket)
            #     host: "my-minio-host:9000"
            #     key: "12345678"
            #     secret: "12345678"
            #     multipart: false
            #     secure: false
            # }
        ]
    }
    boto3 {
        pool_connections: 512
        max_multipart_concurrency: 16
    }
}

Azure Storage¶

Azure Storage credentials - Create an azure.storage.containers section and specify your credentials.

azure.storage {
        containers: [
              {
              # Replace YourAccountName with your Azure Storage account name.
              account_name: "YourAccountName"
              # Replace YourAccountKey with your Azure Storage account key.
              account_key: "YourAccountKey"
              # Container is optional, replace YourContainer with your container name.
              container_name: "YourContainer"
              }
        ]
    }
}

Google Cloud Storage¶

For Google Cloud Storage, you can use default project credentials for all buckets, or specify credentials per individual buckets and sub-directory.

Google Cloud Storage credentials for a project - In the google.storage section, modify clearml.conf by specifying your project and credentials_json file:

google.storage {
        # # Default project and credentials file
        # # Will be used when no bucket configuration is found
        project: "clearml"
        credentials_json: "/path/to/credentials.json"
        # Specific credentials per bucket and sub directory
        # credentials = [
        #     {
        #         bucket: "my-bucket"
        #         subdir: "path/in/bucket" # Not required
        #         project: "clearml"
        #         credentials_json: "/path/to/credentials.json"
        #     },
        # ]
    }

Google Cloud Storage credentials for each bucket and sub-directory, specify a set of credentials as follows (use subdir for sub-directories in your bucket, only):

google.storage {
    # # Default project and credentials file
    # # Will be used when no bucket configuration is found
    # project: "clearml"
    # credentials_json: "/path/to/credentials.json"

    # # Specific credentials per bucket and sub directory
    credentials = [
        {
            bucket: "my-bucket"
            subdir: "path/in/bucket" # Not required
            project: "clearml"
            credentials_json: "/path/to/credentials.json"
        },
    ]
}

Note

Advanced Google Cloud Storage users can configure Google Cloud Storage templates for the ClearML configuration.