"Technology is the answer, but what was the question?"

Find it interesting that this quote of Cedric Price is relatable to many situations in technological problem-solving.

Years ago was part of solutioning on Grid computing using OpenMosix and SunGrid clusters. This is not exactly an HPC - but concepts around distributing the jobs, and queuing are similar. HPC needs jobs to be written in a specific way - more on that in another time.

At a high level; HPC is a cluster responsible for scheduling, dispatching, and managing the remote and distributed execution of number of parallel computing "jobs".
HPC is heavily used for long-running engineering simulation jobs wherein there is stuff like message passing, state synchronizing etc going on between different compute nodes.

When I started Kubernetes journey, given the similar sounding component names (in the clustering concepts of HPC and k8s) at the outset - like control plane, workload distribution etc compelled my hippocampus to an analogy with HPC. However, a Container Orchestrator is very much different. It is made for solving a different set of problems.

An orchestrator's purpose is to run containerized "applications"(not "jobs") on a different set of nodes for better resource utilization - but that's not the main goal of a container orchestrator. Container orchestration system solves problems related to automated software deployment, scaling, and management.

This is vastly different from the need of engineers/scientists who submit high throughput computing jobs to a HPC cluster(mostly via a command line) and wait for results after the computing load, distributed to multiple nodes, does its job.

Saw a few articles on combining workloads of HPC on K8s - but it beats me why would one try to do that? The problems each of these technologies solve are different.

As Robert Sapolsky(a neuroendocrinologist) puts it - we do think in "buckets," - term he uses to refer to conceptual filters people employ that keep them locked in particular worldviews.

How do we avoid this trap (an inclination) to try and use a single solution to each and every problem.. even if its not relevant to the problem ..like say happened in the case of blockchain.