to the systems/facility person, though, they are often not as comfortable, because they tend to behave not quite normally. for instance, non-mainstream hardware often uses proprietary dimms, less conventional BIOS interfaces, IPMI, etc. and there are other non-source-available components (Nvidia GPU drivers, for instance).

I think that teething pains due to the need for tuning software to the newest combinations of hardware sub-components are to be expected when the goal is to extract maximum oomph out of a machine. Aurora (sleeping beauty) is a case in point as it ran at 1/4 of full expected perf 4 months after being kissed into seeing first light (and yet because it is so awesome, that meant 585 PF/s! Wow!). Standardizing on Neoverse V2 may help some, but, as with dragsters, I think there’ll always be a need to tune ops for specific network hardware, memory hierarchies (V-cache, HBM, DDR, CXL 3.0), DPUs and other accelerators, if any.

As luis (River) almost commented in the “Sustainability Headache” piece (11/17/23) “patience and great scientific inspiration”, along with dedicated skunkworks, will “Make great [ARM HPC] (again !!)”.