Deploying ClearML Server: Google Cloud Platform¶
Important
This documentation page applies to deploying your own open source ClearML Server. It does not apply to ClearML Hosted Service users.
Deploy ClearML Server on the Google Cloud Platform (GCP) using use one of the pre-built GCP Custom Images. ClearML provides custom images for each released version of ClearML Server. For a list of the pre-built custom images, see ClearML Server GCP Custom Image.
After deploying ClearML Server, configure the ClearML Python Package for it, see Configuring ClearML for ClearML Server.
Warning
If you are reinstalling ClearML Server, we recommend clearing your browser cookies for ClearML Server. For example, go to Developer Tools > Storage > Cookies (Firefox), Developer Tools > Application > Cookies (Chrome) and deleting all cookies under the ClearML Server URL.
Default ClearML Server service ports¶
After deploying ClearML Server, the services expose the following node ports:
Web server on
8080
API server on
8008
File Server on
8081
Default ClearML Server storage paths¶
The persistent storage configuration:
MongoDB:
/opt/clearml/data/mongo/
Elasticsearch:
/opt/clearml/data/elastic_7/
File Server:
/mnt/fileserver/
Importing the Custom Image to your GCP account¶
Before launching an instance using a ClearML Server GCP Custom Image, import the image to your custom images list.
Note
No upload of the image file is required. We provide links to image files stored in Google Storage.
To import the image to your custom images list:
In the Cloud Console, go to the Images page.
At the top of the page, click Create image.
In Name, specify a unique name for the image.
Optionally, specify an image family for your new image, or configure specific encryption settings for the image.
In the Source menu, select Cloud Storage file.
Enter the ClearML Server image bucket path (see ClearML Server GCP Custom Image), for example:
allegro-files/clearml-server/clearml-server.tar.gz
.Click Create to import the image. The process can take several minutes depending on the size of the boot disk image.
For more information see Import the image to your custom images list in the Compute Engine Documentation.
Launching¶
Warning
By default, ClearML Server with unrestricted access. To restrict ClearML Server access, follow the instructions in the Securing Your Own ClearML Server page.
To launch ClearML Server using a GCP Custom Image, see the Manually importing virtual disks in the “Google Cloud Storage” documentation, Compute Engine documentation. For more information on Custom Images, see Custom Images in the “Compute Engine documentation”.
The minimum requirements for ClearML Server are:
2 vCPUs
7.5GB RAM
Restarting¶
To restart your ClearML Server Docker deployment:
To restart ClearML Server, first stop and then restart the Docker containers by executing the following commands:
docker-compose -f /opt/clearml/docker-compose.yml down docker-compose -f /opt/clearml/docker-compose.yml up -d
Upgrading¶
To upgrade your ClearML Server Docker deployment:
Shut down the docker containers. Execute the following command:
docker-compose -f docker-compose.yml down
If you are upgrading from Trains Server, do the following:
If you are upgrading from Trains Server version 0.15 or older to ClearML Server, a data migration is required before you upgrade. First follow these data migration instructions, and then continue this upgrade.
Rename
/opt/trains
and its subdirectories to/opt/clearml
.sudo mv /opt/trains /opt/clearml
We recommend backing up your data and, if your configuration folder is not empty, backing up your configuration.
Download the latest
docker-compose.yml
file.curl https://raw.githubusercontent.com/allegroai/clearml-server/master/docker/docker-compose.yml -o /opt/clearml/docker-compose.yml
Startup ClearML Server. This automatically pulls the latest ClearML Server build.
docker-compose -f /opt/clearml/docker-compose.yml pull docker-compose -f /opt/clearml/docker-compose.yml up -d
If issues arise during your upgrade, see the FAQ page, How do I fix Docker upgrade errors?.
Backing up and restoring data and configuration¶
The commands in this section are an example of how to backup and restore your data and configuration .
If your data and configuration folders are in /opt/clearml
, then archive all data into ~/clearml_backup_data.tgz
, and your configuration into ~/clearml_backup_config.tgz
:
sudo tar czvf ~/clearml_backup_data.tgz -C /opt/clearml/data .
sudo tar czvf ~/clearml_backup_config.tgz -C /opt/clearml/config .
If you need to restore your data and configuration:
Verify you have the backup files.
Replace any existing data with the backup data:
sudo rm -fR /opt/clearml/data/* /opt/clearml/config/* sudo tar -xzf ~/clearml_backup_data.tgz -C /opt/clearml/data sudo tar -xzf ~/clearml_backup_config.tgz -C /opt/clearml/config
Grant access to the data:
sudo chown -R 1000:1000 /opt/clearml
Examples and use cases¶
For examples and use cases, see the Examples section in this User Guide documentation, and the examples folder of the ClearML repository.
ClearML Server GCP Custom Image¶
The following a section contains a list of Custom Image URLs (exported in different formats) for each released ClearML Server version.