Deploying ClearML Server: Kubernetes Using Helm

Important

This documentation page applies to deploying your own open source ClearML Server. It does not apply to ClearML Hosted Service users.

Warning

If you are reinstalling ClearML Server, we recommend clearing your browser cookies for ClearML Server. For example, go to Developer Tools > Storage > Cookies (Firefox), Developer Tools > Application > Cookies (Chrome) and deleting all cookies under the ClearML Server URL.

Prerequisites

  • A Kubernetes cluster.

  • kubectl is installed and configured (see Install and Set Up kubectl in the Kubernetes documentation).

  • helm is installed (see Installing Helm in the Helm documentation).

  • One node labeled app=clearml.

Warning

ClearML Server deployment uses node storage. If more than one node is labeled as app=clearml, and you redeploy or update later, then ClearML Server may not locate all your data.

Deploying

Warning

By default, ClearML Server with unrestricted access. To restrict ClearML Server access, follow the instructions in the Securing Your Own ClearML Server page.

Step 1: Modify Elasticsearch default values in the Docker configuration file

Before deploying ClearML Server in a Kubernetes cluster, you must modify several Elasticsearch settings in the Docker configuration. For more information, see Install Elasticsearch with Docker in the Elasticsearch documentation and Daemon configuration file in the Docker documentation.

To modify Elasticsearch default values in your Docker configuration file:

  1. Connect to the node in the Kubernetes cluster that you labeled app=clearml.

  2. Create or edit (if one exists) the /etc/docker/daemon.json file, and add or modify the defaults-ulimits section as the following example shows:

     {
         "default-ulimits": {
             "nofile": {
                 "name": "nofile",
                 "hard": 65536,
                 "soft": 1024
             },
             "memlock":
             {
                 "name": "memlock",
                 "soft": -1,
                 "hard": -1
             }
         }
     }
    
  3. Elasticsearch requires that the vm.max_map_count kernel setting, which is the maximum number of memory map areas a process can use, is set to at least 262144.

    For CentOS 7, Ubuntu 16.04, Mint 18.3, Ubuntu 18.04 and Mint 19.x, we tested the following commands to set vm.max_map_count:

     echo "vm.max_map_count=262144" > /tmp/99-clearml.conf
     sudo mv /tmp/99-clearml.conf /etc/sysctl.d/99-clearml.conf
     sudo sysctl -w vm.max_map_count=262144
    
  4. Restart docker:

     sudo service docker restart
    

Step 2. Deploy ClearML Server in the Kubernetes using Helm

After modifying several Elasticsearch settings in your Docker configuration (see Step 1 below), you can deploy ClearML Server.

To deploy ClearML Server in Kubernetes using Helm:

  1. Add the clearml-server repository to your Helm:

     helm repo add allegroai https://allegroai.github.io/clearml-server-helm/
    
  2. Confirm the clearml-server repository is now in Helm:

     helm search clearml
    

    The helm search results must include allegroai/clearml-server-chart.

  3. Install clearml-server-chart on your cluster:

     helm install allegroai/clearml-server-chart --namespace=clearml --name clearml-server
    

    A clearml namespace is created in your cluster and clearml-server is deployed in it.

Port Mapping

After ClearML Server is deployed, the services expose the following:

  • API server on 30008.

  • Web server on 30080.

  • File server on 30081.

The node ports map to the following container ports:

  • 30080 maps to clearml-webserver container on port 8080

  • 30008 maps to clearml-apiserver container on port 8008

  • 30081 maps to clearml-fileserver container on port 8081

Important

We recommend using the container ports (8080, 8008, and 8081), or a load balancer (see the next section, Accessing ClearML Server).

Accessing ClearML Server

To access your ClearML Server:

  • Create a load balancer and domain with records pointing to ClearML Server using the following rules which ClearML uses to translate domain names:

    • The record to access the ClearML Web UI:

        *app.<your domain name>.* 
      

    For example, clearml.app.mydomainname.com points to your node on port 30080.

    • The record to access the ClearML API:

        *api.<your domain name>.* 
      

    For example, clearml.api.mydomainname.com points to your node on port 30008.

    • The record to access the ClearML file server:

        *files.<your domain name>.*
      

      For example, clearmlfiles.mydomainname.com points to your node on port 30081.

Upgrading

Note

We strongly encourage you to keep your ClearML Server up to date, by upgrading to the current release.

  1. Upgrade using new or upgrade values.yaml

     helm upgrade clearml-server allegroai/clearml-server-chart -f new-values.yaml
    
  2. If you previously deployed a ClearML Server, you must first delete old deployments using the following command:

     helm delete --purge clearml-server
    
  3. If you are upgrading from Trains Server version 0.15 or older to ClearML Server, a data migration is required before you upgrade. First follow these data migration instructions, and then continue this upgrade.

  4. Upgrade your deployment to match repository version.

     helm upgrade clearml-server allegroai/clearml-server-chart