An Introduction to HPC

In this video we sit down to discuss HPC – High Performance Computing. What it is, who uses it and what the future looks like for businesses looking to use the technology. 

What is High Performance Computing?

So, High Performance Computing (HPC) is really all about bringing together various compute resources to get workloads done effectively.

I would break down it into two main areas – at least from the way that we would look at it from a YellowDog perspective:

1. Embarrassingly parallel workloads. So, each of the individual tasks can be run independently – they’re automic, they’re recoverable – and you can set up massive HPC clusters to can crunch through that.
2. MPI workloads. So, in this case, rather than thinking of machines, we’re actually thinking of combining CPU, across a multitude of machines, to get work done.

What Does MPI Stand For?

Message Passing Interface.

It’s effectively a really, really fast messaging interface between the different compute resources that exist on a HPC cluster. It’s usually backed with very quick network infrastructure, so that a node can rapidly speak to another one of its volunteer nodes, or its master node.

Who Uses High Performance Computing?

If you look at the history of HPC, it comes from the world where supercomputers were built and housed in massive complexes. So, given the expense of those things, they tended to be used by governments, government agencies, academia, and some very large corporations who had the wealth behind them to actually use the technology.

But what has essentially happened over the years is that, as the ability to plug and play different components from different manufacturers has increased, HPC has been more widely adopted.

It’s used quite extensively in things like Meteorology, Life Sciences, Financial Services, Aircraft Design, Car Design and so forth. So, although it was once associated with very large grids, we’re beginning to hear a lot of new start-ups who are looking for these large, HPC-like clusters to get their workloads done.

What About the Future of HPC?

Well, one of the key movements is cloud is here. And of course cloud has been a lifeline for a lot of companies, including us today working remotely. We are using cloud in our every day lives, to do things like this – to do Zoom chats – as well as to house data etc.

There is a movement that has already started to take those HPC workloads that were normally on-premise and put them in the cloud.

There are a couple of considerations that people need to take into account. I think one of the big ones is the fact that you were previously working on HPC clusters that were always on. They were a fixed size, you had a fixed set of compute that was part of that cluster, but now there are different things to overcome.

In particular, how do you make use of HPC in the cloud in a cost effective way? Making sure the clusters you’re bringing up are appropriate for the workloads you want to use. There are other things as well around things like asynchronicity and heterogeneity.

Asynchronicity is, for example, how do I know if I’m using multiple different types of machine? Or, even multiple providers? When I’m getting status updates, I might get them from one machine at one moment and another machine at another moment.

Heterogeneity is when you’re looking at different types of machines and different types of specifications, and asking yourself, “what is the best compute for my workload?”

As businesses move from the expectations associated with fixed grids, there will be a real focus on how to make these workloads a lot more flexible.