The Power of a Cloud Native Scheduler

This blog discusses how enterprise job schedulers can fall short in cloud environments and how they should be built from the ground up, with the cloud in mind.

Traditional Enterprise Job Schedulers

Often referred to as workload automation, enterprise job scheduling manages task execution on remote servers; resubmitting jobs when there are failures and launching jobs based on any dependencies in the sequence. Traditional enterprise job schedulers offer a graphical user interface (GUI) for this management, and have been used for a number of years for on-premise infrastructure.

Schedulers work in the background, managing multiple threads or pipelines simultaneously and require little thought or intervention. This modus operandi is possible as the number of available nodes and locations are known, and generally owned and stable. Therefore, they can merely be left to wait until work is allocated to them – organisations don’t have to fire them up, provision them or configure them, they can just use them as and when they have a job that needs to be completed.

For a scheduler to work in the cloud, it needs to be cloud native, fault tolerant and scalable.

Cloud Native

While some traditional schedulers can work in the cloud, there will likely always be a ceiling to their performance. Instead of repurposing an existing tool, it is more advantageous to build a scheduler from the ground up, with the cloud in mind. With a truly cloud native tool, its potential will not be limited.

Fault-Tolerant

As instances are often pre-emptible in the cloud, a scheduler needs to be able respond to workers being lost or reclaimed by the provider. It needs to be able to replace workers in the middle of a job, without allowing the entire job to fail and restart. It also needs to be able to intelligently allocate workers based on given criteria, such as a particular region or cloud provider, so the organisation can maintain control over where the work is executed.

Scalable

In order to meet the requirements of compute-intensive workloads at peak times, a scheduler needs to be scalable. It needs to be able to handle thousands of tasks simultaneously, across multiple regions, machine shapes and instance types, for organisations to make the most of the resources available to them.

Customer Value

So what’s the true value of cloud native scheduler?

Designed for the Cloud

The YellowDog Scheduler is designed to work in the cloud. It is equipped to elegantly respond when compute conditions change and has been built to scale.

It also sits on top of, and works with, our Compute Service. Therefore, organisations can dynamically identify where the best workers are – using our intelligent provisioning templates and our Best Source of Compute algorithms – and use our Scheduler Service to automatically place the jobs in that preferential compute.

With an easy-to-use GUI, organisations don’t need to have touch points with multiple providers to execute their workloads. They can view and manage everything from one Platform.