Transformers Broke the Hardware Model

 Attention is elegant — and brutally unfriendly to silicon

Core thesis:
Transformers scaled because of data and parallelism, but attention’s quadratic memory footprint quietly violated every hardware assumption accelerators were built on.

Hardware angle:

• Why attention is memory-bound

• Why KV cache dominates inference cost

• Why FlashAttention mattered more than bigger GPUs

Key insight:

“Attention didn’t just scale models — it exposed hardware’s weakest link.”