Deploying ClearML Server: AWS EC2 AMIs¶
Important
This documentation page applies to deploying your own open source ClearML Server. It does not apply to ClearML Hosted Service users.
Deployment of ClearML Server on AWS is easily performed using AWS AMIs which are available in the AWS Marketplace catalog and in the AWS community AMI catalog.
AWS Marketplace ClearML Server is coming soon - Preconfigured with unique initial access credentials. Until it arrives, use AWS Marketplace Trains Server with the instructions on the page.
ClearML Server community AMIs - Configured by default without authentication to allow quick access and onboarding.
After deploying either type of AMI, you will be able to configure your ClearML Server instance to provide the authentication scheme that best matches your workflow.
Warning
If you are reinstalling ClearML Server, we recommend clearing your browser cookies for ClearML Server. For example, go to Developer Tools > Storage > Cookies (Firefox), Developer Tools > Application > Cookies (Chrome) and deleting all cookies under the ClearML Server URL.
Launching¶
Warning
By default, ClearML Server deploys as an open network. To restrict ClearML Server access, follow the instructions in the Securing Your Own ClearML Server page.
The minimum recommended amount of RAM is 8 GB. For example, a t3.large or t3a.large EC2 instance type would accommodate the recommended RAM size.
AWS community AMIs¶
To launch a ClearML Server AWS community AMI:
Use one of the ClearML Server AWS community AMIs and see:
The AWS Knowledge Center page, How do I launch an EC2 instance from a custom Amazon Machine Image (AMI)?
Detailed instructions in the AWS Documentation for Launching an Instance Using the Launch Instance Wizard.
AWS Marketplace AMIs¶
Launch a ClearML Server AWS Marketplace AMI through the AWS Marketplace website.
To launch a ClearML Server AWS Marketplace AMI:
Open the AWS Marketplace for the Allegro AI ClearML Server.
In the heading area, click Continue to Subscribe.
On the Subscribe to software page, click Accept Terms, and then click Continue to Configuration.
On the Configure this software page, complete the following:
In the Fulfillment Option list, select 64-bit (x86) Amazon Machine Image (AMI).
In the Software Version list, select your ClearML Server version. For example, 0.13.0 (Mar 02, 2020).
In the Region list, select your region.
Click Continue to Launch.
On the Launch this software page, in the Choose Action list, select either of following options, and perform the steps for that option:
Launch through EC2:
Click Launch.
Follow the instructions on the How do I launch an EC2 instance from a custom Amazon Machine Image (AMI)? AWS documentation page.
Launch from Website:
Select your required settings: EC2 Instance Type, VPC Settings, Subnet Settings, Security Group Settings, and Key Pair Settings.
Click Launch.
On the Launch this software page, note your Instance ID. You can use it later to search for your instance in the EC2 Console.
Accessing ClearML Server¶
Once deployed, ClearML Server exposes the following services:
Web server on
TCP port 8080
API server on
TCP port 8008
File Server on
TCP port 8081
To locate your ClearML Server address:
Go to your AWS EC2 Console.
In the Details tab, Public DNS (IPv4) shows your ClearML Server address.
To access ClearML Server Web-App (UI):
Direct your browser to its web server URL:
http://<Server Address>:8080
To SSH into ClearML Server:
Log into the AWS AMI using the default user name
ec2-user
. Control the SSH credentials from the AWS management console.
Logging in to the Web-App (UI)¶
To log in to the ClearML Web-App (UI):
If you launched ClearML Server from an AWS Community AMI, enter any name.
If you launched ClearML Server through the AWS Marketplace, enter the preconfigured default login credentials, which are:
clearml-user (the default user name).
The ClearML Server EC2 instance ID (the default password).
You can modify the default login behavior to match your workflow policy, see Configuring Web Login Authentication on the “Configuring Your Own ClearML Server” page.
Storage configuration¶
The pre-built ClearML Server storage configuration is the following:
MongoDB:
/opt/clearml/data/mongo/
Elasticsearch:
/opt/clearml/data/elastic_7/
File Server:
/mnt/fileserver/
Upgrading¶
Note
For upgrade purposes, the terms Trains Server and ClearML Server are interchangeable.
Upgrading on the same AWS instance¶
This section contains the steps to upgrade ClearML Server on the same AWS instance.
Warning
Some legacy Trains Server AMIs provided an auto-upgrade on restart capability. This functionality is now deprecated.
To upgrade your ClearML Server AWS AMI:
Shutdown the ClearML Server executing the following command (which assumes the configuration file is in the environment path).
docker-compose -f /opt/clearml/docker-compose.yml down
If you are upgrading from Trains Server, use this command:
docker-compose -f /opt/trains/docker-compose.yml down
We recommend backing up your data and, if your configuration folder is not empty, backing up your configuration.
If you are upgrading from Trains Server version 0.15 or older, a data migration is required before you upgrade. First follow these data migration instructions, and then continue this upgrade.
If you are upgrading from Trains Server to ClearML Server, rename
/opt/trains
to/opt/clearml
.Download the latest
docker-compose.yml
file. Execute the following command:sudo curl https://raw.githubusercontent.com/allegroai/clearml-server/master/docker/docker-compose.yml -o /opt/clearml/docker-compose.yml
Startup ClearML Server. This automatically pulls the latest ClearML Server build.
docker-compose -f /opt/clearml/docker-compose.yml pull docker-compose -f docker-compose.yml up -d
Upgrading and migrating to a new AWS instance¶
This section contains the steps to upgrade ClearML Server on the new AWS instance.
To migrate and upgrade your ClearML Server AWS AMI:
Shutdown ClearML Server. Executing the following command (which assumes the configuration file is in the environment path).
docker-compose down
On the old AWS instance, backup your data and, if your configuration folder is not empty, backup your configuration.
If you are upgrading from Trains Server version 0.15 or older, a data migration is required before you upgrade. First follow these data migration instructions, and then continue this upgrade.
On the new AWS instance, restore your data and, if your configuration folder is not empty, restore your configuration.
Startup ClearML Server. This automatically pulls the latest ClearML Server build.
docker-compose -f docker-compose.yml pull docker-compose -f docker-compose.yml up -d
Backing up and restoring data and configuration¶
Note
If you’re moving data between a Trains Server and a ClearML Server installation, notice using the correct paths for backup and restore (/opt/trains and /opt/clearml respectively).
The commands in this section are examples for backing up and restoring your data and configuration.
If your data and configuration folders are in /opt/clearml
, then archive all data into ~/clearml_backup_data.tgz
, and your configuration into ~/clearml_backup_config.tgz
:
sudo tar czvf ~/clearml_backup_data.tgz -C /opt/clearml/data .
sudo tar czvf ~/clearml_backup_config.tgz -C /opt/clearml/config .
If you need to restore your data and configuration:
Verify you have the backup files.
Replace any existing data with the backup data:
sudo rm -fR /opt/clearml/data/* /opt/clearml/config/* sudo tar -xzf ~/clearml_backup_data.tgz -C /opt/clearml/data sudo tar -xzf ~/clearml_backup_config.tgz -C /opt/clearml/config
Grant access to the data:
sudo chown -R 1000:1000 /opt/clearml
ClearML Server AWS community AMIs¶
The following sections contain lists of AMI Image IDs, per region, for each released ClearML Server version.
Latest version¶
v0.17.0¶
eu-north-1 : ami-0423debdbd76e19bc
ap-south-1 : ami-045676710ca7be441
eu-west-3 : ami-0c27df66df489e7fe
eu-west-2 : ami-006f3e468e893fc99
eu-west-1 : ami-05636fed69e4a40ba
ap-northeast-2 : ami-03fa1ab9929e1ce79
ap-northeast-1 : ami-09472b008b38e01eb
sa-east-1 : ami-000403050edd4aa76
ca-central-1 : ami-0dd7ced864cd6b2b3
ap-southeast-1 : ami-0cfdd5eae2e8fd132
ap-southeast-2 : ami-0f07d66903628d1c8
eu-central-1 : ami-039a50823e5472523
us-east-2 : ami-09a01600b3d781a23
us-west-1 : ami-09dd4d75dc245edb6
us-west-2 : ami-0d665dcb4e6d22f58
us-east-1 : ami-0d9fbf285df0b3067