HPC – There’s plenty of room at the bottom

Mike Croucher

EPSRC Research Software Engineering Fellow

Co-founder of RSE-Sheffield

HPC

Geek Top Gear

Any questions?

• Audience member 1: What’s a core?
• Audience member 2: Why does it run my R script slower than my laptop?
• Audience member 3: Do you have Excel installed on it?

What we like

What they need

Traditional HPC is irrelevant to most researchers!

...and most of the rest use it badly.

Hannay et al (2009)

• Online survey of 1972 international researchers
• ~80% never use a supercomputer

Prabhu et al (2009)

• Interviewed 114 researchers at Princeton
• ~40% never use a supercomputer

Prabhu et al (2009)

• "Despite enormous wait times, many scientists run their programs only on desktops"
• "About a third of researchers did not use any form of parallelism in their research at all"
• “Currently, many researchers fit their scientific models to only a subset of available parameters for faster program runs.”

Prabhu et al (2009)

“Across disciplines, an order of magnitude performance improvement was cited as a requirement for significant changes in research quality”

10x speedup? No problem!

RSE Support + HPC / Cloud

Research Software Engineers

RSE Sheffield

• Collaborate with researchers
• Collaborate with HPC / Cloud providers

HPC: What does RSE need?

• A compute substrate
• that caters for the long tail
• that's flexible
• that's easy to manage

Traditional HPC

• A compute substrate (YES)
• that caters for the long tail (PARTIALLY)
• that's flexible (NO)
• that's easy to manage (NO)

Biggest problem

Lack of flexibility

Cloud

• A compute substrate (YES)
• that caters for the long tail (YES)
• that's flexible (YES)
• that's easy to manage (NO)

Biggest problem

Billing management!

Future of academic HPC

Interaction with Research Software Engineering