Deploying ClearML Server: Linux and macOS

Important

This documentation page applies to deploying your own open source ClearML Server. It does not apply to ClearML Hosted Service users.

Deploy the ClearML Server in Linux or macOS using the pre-built Docker image.

For ClearML docker images, including previous versions, see https://hub.docker.com/r/allegroai/clearml. However, pulling the ClearML Docker image directly is not required. We provide a docker compose YAML file that does this for you. The docker compose file is included in the instructions on this page.

Warning

If you are reinstalling ClearML Server, we recommend clearing your browser cookies for ClearML Server. For example, go to Developer Tools > Storage > Cookies (Firefox), Developer Tools > Application > Cookies (Chrome) and deleting all cookies under the ClearML Server URL.

Prerequisites

For Linux users only:

  • Your Linux distribution must support Docker. For more information, see this explanation in the Docker documentation.

  • You must be logged in as a user with sudo privileges.

  • Use bash for all command-line instructions in this installation.

  • The ports 8080, 8081, and 8008 must be available for the ClearML Server services.

Deploying

Warning

By default, ClearML Server with unrestricted access. To restrict ClearML Server access, follow the instructions in the Securing Your Own ClearML Server page.

To launch your ClearML Server on Linux or macOS:

  1. Install Docker. The instructions depend upon your operating system:

  2. Verify the Docker CE installation. Execute the command:

     docker run hello-world
    

    The expected is output is:

     Hello from Docker!
     This message shows that your installation appears to be working correctly.
     To generate this message, Docker took the following steps:
    
     1. The Docker client contacted the Docker daemon.
     2. The Docker daemon pulled the "hello-world" image from the Docker Hub. (amd64)
     3. The Docker daemon created a new container from that image which runs the executable that produces the output you are currently reading.
     4. The Docker daemon streamed that output to the Docker client, which sent it to your terminal.
    
  3. For macOS, only, increase the memory allocation in Docker Desktop to 8GB.

    1. In your top status bar, click the Docker icon.

    2. Click Preferences, Resources, Advanced, and then set the memory to at least 8192.

    3. Click Apply.

  4. For Linux, only, install docker-compose. Execute the following commands (for more information, see Install Docker Compose in the Docker documentation):

     sudo curl -L "https://github.com/docker/compose/releases/download/1.24.1/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
     sudo chmod +x /usr/local/bin/docker-compose
    
  5. Increase vm.max_map_count for Elasticsearch in Docker. Execute the following commands, depending upon your operating system:

    • Linux:

        echo "vm.max_map_count=262144" > /tmp/99-clearml.conf
        sudo mv /tmp/99-clearml.conf /etc/sysctl.d/99-clearml.conf
        sudo sysctl -w vm.max_map_count=262144
        sudo service docker restart
      
    • macOS:

        screen ~/Library/Containers/com.docker.docker/Data/vms/0/tty
        sysctl -w vm.max_map_count=262144
      
  6. Remove any previous installation of ClearML Server.

    This clears all existing ClearML SDK databases.

     sudo rm -R /opt/clearml/
    
  7. Create local directories for the databases and storage.

     sudo mkdir -p /opt/clearml/data/elastic_7
     sudo mkdir -p /opt/clearml/data/mongo/db
     sudo mkdir -p /opt/clearml/data/mongo/configdb
     sudo mkdir -p /opt/clearml/data/redis
     sudo mkdir -p /opt/clearml/logs
     sudo mkdir -p /opt/clearml/config
     sudo mkdir -p /opt/clearml/data/fileserver
    
  8. For macOS, only, do the following:

    1. Open the Docker app.

    2. Select Preferences.

    3. On the File Sharing tab, add /opt/clearml.

  9. Grant access to the Dockers, depending upon your operating system.

    • Linux:

        sudo chown -R 1000:1000 /opt/clearml
      
    • macOS:

        sudo chown -R $(whoami):staff /opt/clearml
      
  10. Download the ClearML Server docker-compose YAML file.

     sudo curl https://raw.githubusercontent.com/allegroai/clearml-server/master/docker/docker-compose.yml -o /opt/clearml/docker-compose.yml
    
  11. For Linux only, configure the ClearML Agent Services. If CLEARML_HOST_IP is not provided, then ClearML Agent Services will use the external public address of the ClearML Server. If CLEARML_AGENT_GIT_USER / CLEARML_AGENT_GIT_PASS are not provided, then ClearML Agent Services will not be able to access any private repositories for running service tasks.

     export CLEARML_HOST_IP=server_host_ip_here
     export CLEARML_AGENT_GIT_USER=git_username_here
     export CLEARML_AGENT_GIT_PASS=git_password_here
    
  12. Run docker-compose with the downloaded configuration file.

     docker-compose -f /opt/clearml/docker-compose.yml up -d
    

Your server is now running on http://localhost:8080.

Port mapping

After deploying ClearML Server, the services expose the following ports:

  • Web server on port 8080

  • API server on port 8008

  • File server on port 8081

Restarting

To restart your ClearML Server Docker deployment:

  • To restart ClearML Server, first stop and then restart the Docker containers by executing the following commands:

      docker-compose -f /opt/clearml/docker-compose.yml down
      docker-compose -f /opt/clearml/docker-compose.yml up -d
    

Upgrading

Important: Upgrading from v0.14 or older

For Linux only, if you are upgrading from Trains Server v0.14 or older, configure the ClearML Agent Services.

  • If CLEARML_HOST_IP is not provided, then ClearML Agent Services will use the external public address of the ClearML Server.

  • If CLEARML_AGENT_GIT_USER / CLEARML_AGENT_GIT_PASS are not provided, then ClearML Agent Services will not be able to access any private repositories for running service tasks.

export CLEARML_HOST_IP=server_host_ip_here
export CLEARML_AGENT_GIT_USER=git_username_here
export CLEARML_AGENT_GIT_PASS=git_password_here

Note

For backwards compatibility, the environment variables TRAINS_HOST_IP, TRAINS_AGENT_GIT_USER, and TRAINS_AGENT_GIT_PASS are supported.

To upgrade your ClearML Server Docker deployment:

  1. Shutdown ClearML Server. Executing the following command (which assumes the configuration file is in the environment path).

     docker-compose -f docker-compose.yml down
    
  2. If you are upgrading from Trains Server version 0.15 or older to ClearML Server, a data migration is required before you upgrade. First follow these data migration instructions, and then continue this upgrade.

  3. We recommend backing up your data and, if your configuration folder is not empty, backing up your configuration.

  4. If you are upgrading from Trains Server to ClearML Server, rename /opt/trains and its subdirectories to /opt/clearml.

      sudo mv /opt/trains /opt/clearml
    
  5. Download the latest docker-compose.yml file.

     curl https://raw.githubusercontent.com/allegroai/clearml-server/master/docker/docker-compose.yml -o /opt/clearml/docker-compose.yml
    
  6. Startup ClearML Server. This automatically pulls the latest ClearML Server build.

     docker-compose -f /opt/clearml/docker-compose.yml pull
     docker-compose -f /opt/clearml/docker-compose.yml up -d
    

If issues arise during your upgrade, see the FAQ page, How do I fix Docker upgrade errors?.

Backing up and restoring data and configuration

The commands in this section are an example of how to backup and restore your data and configuration .

If your data and configuration folders are in /opt/clearml, then archive all data into ~/clearml_backup_data.tgz, and your configuration into ~/clearml_backup_config.tgz:

sudo tar czvf ~/clearml_backup_data.tgz -C /opt/clearml/data .
sudo tar czvf ~/clearml_backup_config.tgz -C /opt/clearml/config .

If you need to restore your data and configuration:

  1. Verify you have the backup files.

  2. Replace any existing data with the backup data:

     sudo rm -fR /opt/clearml/data/* /opt/clearml/config/*
     sudo tar -xzf ~/clearml_backup_data.tgz -C /opt/clearml/data
     sudo tar -xzf ~/clearml_backup_config.tgz -C /opt/clearml/config 
    
  3. Grant access to the data, depending upon your operating system:

    • Linux:

        sudo chown -R 1000:1000 /opt/clearml
      
    • macOS:

        sudo chown -R $(whoami):staff /opt/clearml