automation.aws_auto_scaler.AwsAutoScaler

class trains.automation.aws_auto_scaler.AwsAutoScaler
spin_down_worker(instance_id)

Destroys the cloud instance.

Parameters

instance_id (str) – Cloud instance ID to be destroyed (currently, only AWS EC2 is supported).

spin_up_worker(resource, worker_id_prefix, queue_name)

Creates a new worker for trains. First, create an instance in the cloud and install some required packages. Then, define trains-agent environment variables and run trains-agent for the specified queue.

Note

  • Wait until instance is running.

  • This implementation assumes the instance image already has docker installed.

Parameters
  • resource (str) – Resource name, as defined in BUDGET and QUEUES.

  • worker_id_prefix (str) – Worker name prefix.

  • queue_name (str) – trains queue to listen to.

supervisor()

Spin up or down resources as necessary. For every queue in self.queues do the following:

  1. Check if there are tasks waiting in the queue.

  2. Check if there are enough idle workers available for those tasks.

  3. In case more instances are required, and we haven’t reached max instances allowed, create the required instances with regards to the maximum number defined in self.queues Choose which instance to create according to their order in self.queues. Won’t create more instances if maximum number defined has already reached.

Spin down instances according to their idle time. instance which is idle for more than self.max_idle_time_min minutes would be removed.