Trains Python Package Extras

Trains is now ClearML

This documentation applies to the legacy Trains versions. For the latest documentation, see ClearML.

This page describes how to install and configure Trains Python Package Extras. Currently, the Trains Python Package Extras include extras for experiment storage of model checkpoints (snapshots) and other artifacts. The supported Cloud storage types include:

AWS S3
Azure Storage
Google Cloud Storage

In addition to installing a Trains extra for storage, you must add your storage bucket names, credentials, and related information to the Trains configuration file.

Step 1. Install Trains extras for storage

To install a Trains extra for storage, do the following:

For your Cloud storage, execute one of the following commands:

AWS S3
```
pip install trains[s3]
```
Google Cloud Storage
```
pip install trains[gs]
```
Azure Storage
```
pip install trains[azure]
```

Step 2. Initializing a new Trains configuration file

Use our trains-init setup wizard to initialize a new Trains configuration file.

If you already have a configuration file, the setup wizard will notify you. You can then skip to the Step 3. Add storage credentials to your Trains configuration file, where you add Trains settings to an existing Trains configuration file.

Why would a Trains configuration file already exist?

If you previously configured Trains for the demo Trains Server, a self-hosted Trains Server, or installed and configure Trains Agent, you may have an existing configuration file. This is because Trains and Trains Agent use the same configuration file, but not all the same configuration options.

To initialize a new Trains configuration file, do the following:

Open a terminal session.
In your terminal session, run the setup wizard.
```
trains-init
```
If the setup wizard's response indicates you already have a configuration file, then go to Step 3. Add storage credentials to your Trains configuration file and add your storage credentials. The wizard does not edit or overwrite existing configuration files.
Additional Trains configuration files

You can create additional Trains configuration files using the --file option of trains-init.

For example, create a configuration file to use inside Docker containers when executing a Task:
```
trains-init --file MyOtherTrains.conf
```
and then specify it using the TRAINS_CONFIG_FILE environment variable inside the container:
```
TRAINS_CONFIG_FILE = MyOtherTrains.conf
```
For more information about running experiments inside Docker containers, see Trains Agent Use Case Examples and Trains Agent Reference pages.
The setup begins.
```
TRAINS SDK setup process

Please create new trains credentials through the profile page in your trains web app (e.g. http://localhost:8080/profile)
In the profile page, press "Create new credentials", then press "Copy to clipboard".

Paste copied configuration here:
```
At the command prompt Paste copied configuration here:, copy and paste the Trains credentials you create as follows:
1. Open the Trains Web-App (UI) in your browser (if you are using are demo Trains Server, go to https://demoapp.trains.allegro.ai/dashboard).
2. Click the PROFILE page.
3. Click Create new credentials.
4. Click Copy to clipboard.
5. In your terminal session, paste your credentials and press Enter.

Enter the API server (press Enter to accept the default value which is based on your Trains credentials):

Detected credentials key="********************" secret="*******"
API Host configured to: [https://demoapi.trains.allegro.ai]

Enter your Trains Server web server (press Enter to accept the default value which is based on your previous response):

demoapi.trains.allegro.ai is the api server, we need the web server. Replacing 'demoapi.' with 'demoapp.'
Web Application Host configured to: [https://demoapp.trains.allegro.ai]

Enter your Trains file server (press Enter to accept the default value which is based on your previous response):
```
File Store Host configured to: [https://demofiles.trains.allegro.ai]
```

Your Trains configuration file is verified and created.

TRAINS Hosts configuration:
Web App: https://demoapp.trains.allegro.ai
API: https://demoapi.trains.allegro.ai
File Store: https://demofiles.trains.allegro.ai

Verifying credentials ...
Credentials verified!

New configuration stored in /home/<username>/trains.conf
TRAINS setup completed successfully.

The configuration file's location depends upon your operating system:

Linux - ~/trains.conf`
Mac - $HOME/trains.conf
Windows - \User\<username>\trains.conf

Step 3. Add storage credentials to your Trains configuration file

AWS S3

For AWS S3, you can use default credentials for all buckets, or specify credentials per individual buckets.

See the AWS documentation for restrictions and limitations on bucket naming.

S3 default credentials - In your configuration file aws.s3 section, insert your key, secret, and region default credentials.

aws {
    s3 {
        # S3 credentials, used for read/write access by various SDK elements

        # default, used for any bucket not specified below        
        key: "my-access-key"
        secret: "my-secret-key"
        region: "my-region"
        credentials: [
            # specifies key/secret credentials to use when handling s3 urls (read or write)
            # {
            #     bucket: "my-bucket-name"
            #     key: "my-access-key"
            #     secret: "my-secret-key"
            # },
            # {
            #     # This will apply to all buckets in this host (unless key/value is specifically provided for a given bucket)
            #     host: "my-minio-host:9000"
            #     key: "12345678"
            #     secret: "12345678"
            #     multipart: false
            #     secure: false
            # }
        ]
    }
    boto3 {
        pool_connections: 512
        max_multipart_concurrency: 16
    }
}

S3 credentials per individual bucket - In the credentials section, uncomment the lines for a set of bucket
credentials and insert your bucket, key, and secret.

aws {
    s3 {
        # S3 credentials, used for read/write access by various SDK elements

        # default, used for any bucket not specified below
        key: ""
        secret: ""
        region: ""

        credentials: [
            # specifies key/secret credentials to use when handling s3 urls (read or write)        
            {
                bucket: "my-bucket-name"
                key: "my-access-key"
                secret: "my-secret-key"
            },
            # {
            #     # This will apply to all buckets in this host (unless key/value is specifically provided for a given bucket)
            #     host: "my-minio-host:9000"
            #     key: "12345678"
            #     secret: "12345678"
            #     multipart: false
            #     secure: false
            # }
        ]
    }
    boto3 {
        pool_connections: 512
        max_multipart_concurrency: 16
    }
}

Azure Storage

Azure Storage credentials - Create an azure.storage.containers section and specify your credentials.

azure.storage {
        containers: [
              {
              # Replace YourAccountName with your Azure Storage account name.
              account_name: "YourAccountName"
              # Replace YourAccountKey with your Azure Storage account key.
              account_key: "YourAccountKey"
              # Container is optional, replace YourContainer with your container name. 
              container_name: "YourContainer"
              }
        ]
    }
}

Google Cloud Storage

For Google Cloud Storage, you can use default project credentials for all buckets, or specify credentials per individual buckets and sub-directory.

Google Cloud Storage credentials for a project - In the google.storage section, modify trains.conf by specifying your project and credentials_json file:

google.storage {
        # # Default project and credentials file
        # # Will be used when no bucket configuration is found        
        project: "trains"
        credentials_json: "/path/to/credentials.json"
        # Specific credentials per bucket and sub directory
        # credentials = [
        #     {
        #         bucket: "my-bucket"
        #         subdir: "path/in/bucket" # Not required
        #         project: "trains"
        #         credentials_json: "/path/to/credentials.json"
        #     },
        # ]
    }

Google Cloud Storage credentials for each bucket and sub-directory, specify a set of credentials as follows (use subdir for sub-directories in your bucket, only):

google.storage {
    # # Default project and credentials file
    # # Will be used when no bucket configuration is found
    # project: "trains"
    # credentials_json: "/path/to/credentials.json"

    # # Specific credentials per bucket and sub directory        
    credentials = [
        {
            bucket: "my-bucket"
            subdir: "path/in/bucket" # Not required
            project: "trains"
            credentials_json: "/path/to/credentials.json"
        },
    ]
}

Advanced Google Cloud Storage users

Advanced users can configure Google Cloud Storage templates for the Trains configuration.