Chapter 2 Why Everything in Deep Learning Becomes Matrix Multiplication

 

Chapter 2

Why Everything in Deep Learning Becomes Matrix Multiplication

The Promise of This Chapter

After this chapter, readers will understand:

• Why CNNs, Transformers, and MLPs all look different — but run the same

• Why accelerators revolve around GEMM

• Why “matrix multiply engines” dominate AI chips

This is where software abstractions collapse into hardware reality.

---

2.1 The Illusion of Diversity in Neural Networks

From the outside, models look very different:

• Convolutions for images

• Attention for language

• MLPs everywhere

But hardware doesn’t see “layers” or “tokens.”

Hardware sees:

Regular, dense numerical operations over large arrays

Once performance matters, all roads lead to matrix multiplication.

---

2.2 Why Matrix Multiply Is the Perfect Hardware Workload

Matrix multiplication has three properties hardware loves:

1. Massive data reuse

   • Each value is used many times

2. Regular access patterns

   • Predictable, schedulable

3. High arithmetic intensity

   • Lots of math per byte moved

On the Roofline graph, GEMM lives far to the right.

This is not a software choice.
It is a physical inevitability.

---

2.3 Convolutions → Matrix Multiply

A convolution:

• Slides a small filter over an image

• Accumulates dot products

By rearranging data (e.g., im2col):

• Each convolution becomes a matrix multiply

• Filters become one matrix

• Image patches become another

This costs some extra memory — but buys:

• Better reuse

• Higher performance

• Accelerator compatibility

Hardware prefers wasteful memory to wasted bandwidth.

---

2.4 Attention → Matrix Multiply

Transformers look exotic, but attention boils down to:

1. Q × Kᵀ

2. Softmax

3. Result × V

Steps (1) and (3) are matrix multiplications.

This is why:

• Attention scales well on GPUs

• TPUs excel at large sequence lengths

• Optimizing attention = optimizing GEMM

---

2.5 MLPs Were Always Matrix Multiply

Fully connected layers are literally:

y = Wx + b

No transformation required.

This is why:

• MLPs scale effortlessly

• They dominate compute cost

• They map cleanly to hardware

---

2.6 Why Accelerators Are “GEMM Machines”

Because:

• GEMM maximizes arithmetic intensity

• GEMM minimizes off-chip traffic

• GEMM maps cleanly to systolic arrays and SIMD

So accelerators are designed around:

• MAC arrays

• Tensor cores

• Systolic grids

Everything else is glue logic.

---

Chapter 2 Takeaway

Deep learning does not run on layers or graphs.
It runs on matrix multiplication.

Once you accept this, accelerator design becomes obvious.

---

For Readers Who Want to Go Deeper 🔍

🟢 Conceptual

• Goodfellow et al. — Deep Learning

• Sze et al. — Efficient Processing of DNNs

🟡 Architecture-Level

• Jouppi et al. — TPU paper

• NVIDIA Tensor Core documentation

🔴 Implementation-Level

• Eyeriss paper

• Gemmini (RISC-V accelerator)

---

Interlude

A Concrete Roofline Example (With Numbers)

Let’s make the Roofline real.

Example Hardware

• Peak compute: 100 TFLOPs

• Memory bandwidth: 1 TB/s

This means:

• To fully use compute, you need
  100 FLOPs per byte

---

Workload A: Vector Add

• ~1 FLOP per byte

• Performance limited to:
  1 TB/s × 1 FLOP/B = 1 TFLOP

➡ Uses 1% of peak compute
➡ Completely memory-bound

---

Workload B: Matrix Multiply

• ~200 FLOPs per byte

• Performance limited to:
  100 TFLOPs (compute roof)

➡ Fully utilizes hardware
➡ Compute-bound

---

Moral

Hardware speedups only matter if your workload moves right on the Roofline.

---

Canonical Roofline Figure (Use This Once Everywhere)

Figure Requirements

• X-axis: Arithmetic Intensity (log scale)

• Y-axis: Performance

• One slanted bandwidth line

• One flat compute roof

• Two dots:

  • “Vector Add” (left)

  • “Matrix Multiply” (right)

This single figure explains:

• GPUs

• TPUs

• Scaling limits

• Why optimizations work

---

Chapter 1 The Roofline Model: The One Graph That Explains AI Hardware

 

Chapter 1

The Roofline Model: The One Graph That Explains AI Hardware

The Promise of This Chapter

If you understand this chapter, you will:

• Instantly see why hardware underperforms

• Understand why FLOPs alone are meaningless

• Predict when GPUs or TPUs will help — and when they won’t

Everything else in this primer builds on this.

---

1.1 Why Peak Performance Numbers Lie

Hardware specs love big numbers:

• “100 TFLOPs”

• “1.5 TB/s bandwidth”

• “Thousands of cores”

But real workloads almost never reach peak performance.

Why?

Because performance is limited by two things, not one:

1. How fast you can compute

2. How fast you can move data

The Roofline model puts both on a single graph.

---

1.2 The Two Limits That Matter

Limit 1: Compute Throughput

This is the best-case scenario:

• Data is already available

• Compute units are fully utilized

• Nothing is waiting

This is the flat roof of the roofline.

Limit 2: Memory Bandwidth

This is the common case:

• Compute units wait for data

• Memory can’t feed them fast enough

This is the slanted roof.

Whichever limit you hit first determines performance.

---

1.3 Arithmetic Intensity: The Key Quantity

The Roofline model introduces one crucial idea:

Arithmetic Intensity = Computation / Data Movement

In simple terms:

• How many math operations do you perform per byte fetched?

Examples:

• Low intensity: vector add (touch data once)

• High intensity: matrix multiply (reuse data many times)

This single number determines whether you are:

• Memory-bound (data-starved)

• Compute-bound (compute-limited)

---

1.4 The Roofline Graph (Conceptually)

Think of the graph like this:

• X-axis: Arithmetic intensity (reuse)

• Y-axis: Performance

There are two regions:

• Slanted line → memory-bound

• Flat line → compute-bound

Your workload moves right as you improve data reuse.

---

1.5 Why Matrix Multiply Is Special

Matrix multiplication sits far to the right:

• Each input is reused many times

• Data movement is amortized

• Arithmetic intensity is high

This is why:

• GPUs shine on GEMM

• TPUs are built around GEMM

• Deep learning maps everything to GEMM

The Roofline doesn’t prefer matrix multiply — physics does.

---

1.6 Why Most Code Is Memory-Bound

Many operations:

• Touch data once

• Do little math

• Move on

Examples:

• Elementwise ops

• Reductions

• Poorly tiled kernels

These sit on the left side of the roofline.

No amount of extra compute fixes this.
Only more reuse does.

---

1.7 How Hardware Designers Use the Roofline

Hardware architects ask:

• Where will real workloads land?

• How far left is memory-bound?

• How expensive is data movement?

This leads directly to:

• Large on-chip SRAMs

• High-bandwidth memory (HBM)

• Systolic arrays

• Massive threading

The roofline explains why these features exist.

---

1.8 GPUs Through the Roofline Lens

GPUs assume:

• Many workloads are memory-bound

• Latency is unavoidable

So they:

• Run thousands of threads

• Switch work while waiting for data

• Use caches and shared memory

GPUs hide memory latency.

---

1.9 TPUs Through the Roofline Lens

TPUs assume:

• Data movement must be minimized

• Workloads are predictable

So they:

• Use fixed dataflow

• Keep data stationary

• Maximize on-chip reuse

TPUs avoid memory latency.

Same roofline. Different strategy.

---

1.10 The Most Important Insight

Here is the insight to underline:

Performance improves more by increasing arithmetic intensity than by increasing peak FLOPs.

This is why:

• Algorithm design matters

• Compiler scheduling matters

• Memory layout matters more than ALU count

---

Chapter 1 Takeaway

If you remember one thing:

The Roofline model tells you why performance stops scaling — and what kind of hardware can fix it.

Before asking:

• “Is this GPU fast enough?”

Ask:

• “Where does my workload sit on the roofline?”

---

For Readers Who Want to Go Deeper 🔍

🟢 Conceptual

• Hennessy & Patterson — Computer Architecture: A Quantitative Approach (Roofline model)

• Williams et al. — Roofline: An Insightful Visual Performance Model

🟡 Architecture-Level

• NVIDIA CUDA Programming Guide (memory hierarchy)

• Jouppi et al. — In-Datacenter Performance Analysis of a TPU

🔴 Hardware / Circuit-Level

• Rabaey — Digital Integrated Circuits

• Chandrakasan — Low Power CMOS Design

---

correctness checking of kernel with floats

 https://deep-reinforce.com/correctness_check

Compute Is Cheap, Data Is Expensive - explore with chatgpt

 

**“Compute Is Cheap, Data Is Expensive:

A Practical Primer on GPUs, TPUs, and AI Hardware”**

Why this works

• Instantly communicates the core insight

• Memorable and quotable

• Signals systems-level thinking (engineers love this)

• Non-intimidating for newcomers

This title alone explains why the hardware exists.

---

Strong Alternatives (Depending on Tone)

“Why AI Hardware Looks the Way It Does”

Subtitle: From Matrix Multiplication to GPUs and TPUs

✔ Excellent for broad audiences
✔ Very “TPU Deep Dive”-like

---

**“From MatMul to Megawatts:

Understanding GPUs, TPUs, and Scaling AI”**

✔ Slightly more dramatic
✔ Good if you want to emphasize scaling and systems

---

“The Shape of AI Hardware”

Subtitle: How Workloads, Memory, and Dataflow Define GPUs and TPUs

✔ Elegant
✔ Feels like a classic systems text

---

More Academic / Engineer-Focused

**“Architectures for Deep Learning:

A Systems-Level Primer”**

✔ Safe, formal
✔ Good if used in classes or internal docs

---

🚫 Titles to Avoid

• “Introduction to GPUs and TPUs” (too generic)

• “Deep Learning Accelerators Explained” (sounds shallow)

• “Modern AI Hardware” (says nothing)

---

2️⃣ How to Add Deep References Without Scaring Beginners

The trick: layered references.

Rule:

Never interrupt the narrative with citations.
Put depth behind collapsible or end-of-chapter references.

---

3️⃣ Reference Structure That Works

At the End of Each Chapter, Add:

🔍 “For Readers Who Want to Go Deeper”

Organize references by intent, not by type.

---

Example: Chapter 0 References

🔧 Hardware & Architecture

• Hennessy & Patterson — Computer Architecture: A Quantitative Approach
  → Roofline model, memory bottlenecks

• Weste & Harris — CMOS VLSI Design
  → Energy cost of data movement

• Rabaey — Digital Integrated Circuits

---

⚙️ AI Accelerators & GPUs

• Jouppi et al. — In-Datacenter Performance Analysis of a TPU

• NVIDIA GPU Architecture Whitepapers

• Kirk & Hwu — Programming Massively Parallel Processors

---

🧠 ML Workloads

• Goodfellow et al. — Deep Learning

• Sze et al. — Efficient Processing of Deep Neural Networks

• Vaswani et al. — Attention Is All You Need

---

📊 Scaling & Systems

• “How to Scale Your Model” (OpenAI / DeepMind-style blog)

• Goyal et al. — Accurate, Large Minibatch SGD

• Megatron-LM & GPipe papers

---

4️⃣ Global “Further Reading” Section (End of Primer)

This is where you earn hardware engineer credibility.

Core Architecture

• Hennessy & Patterson

• ISCA / MICRO / HPCA proceedings

Accelerator Design

• Eyeriss

• Gemmini

• NVDLA

Physical Design & Power

• Chandrakasan — Low Power Digital CMOS

• ISSCC papers

Industry Docs

• NVIDIA CUDA Programming Guide

• Google TPU System Architecture docs

• AMD CDNA whitepapers

---

5️⃣ Subtle but Powerful Trick

Add icons to references:

• 🟢 Conceptual

• 🟡 Architecture-level

• 🔴 Circuit / RTL-level

This tells readers how deep each reference goes without words.

---

6️⃣ My Strong Recommendation

If you want this to become a canonical reference:

Title:

Compute Is Cheap, Data Is Expensive
A Practical Primer on GPUs, TPUs, and AI Hardware

Structure:

• Story-driven chapters

• End-of-chapter deep dives

• One killer idea per chapter

• No inline citations 

Chapter 0 Why Deep Learning Needs Special hardware

 

Chapter 0

Why Deep Learning Needs Special Hardware

The One-Sentence Problem

Modern AI is limited not by how fast we can compute, but by how fast we can move data.

Everything else in this primer follows from that.

---

0.1 Why CPUs Were Enough — Until They Weren’t

For decades, computers were designed for:

• Branch-heavy code

• Small working datasets

• Sequential execution

CPUs are amazing at:

• Running operating systems

• Handling unpredictable control flow

• Doing a little bit of everything

But deep learning is the opposite.

---

0.2 What Deep Learning Actually Does

Under the hood, training and inference mostly repeat one operation:

Multiply large matrices and add the results

Whether it’s:

• Image recognition

• Speech

• Translation

• Chatbots

They all reduce to dense linear algebra.

There are very few branches.
There is enormous repetition.
The same data is reused again and again.

This is the key mismatch with CPUs.

---

0.3 Compute Is Cheap. Data Movement Is Not.

A useful mental rule:

Moving data costs 10–100× more energy than computing on it.

• A multiply-add is cheap

• Fetching data from memory is expensive

• Fetching data from far-away memory is very expensive

As models grow:

• Parameters no longer fit in caches

• Memory bandwidth becomes the bottleneck

• Adding more ALUs stops helping

This is why “just faster CPUs” failed.

---

0.4 The Hidden Enemy: Memory Bandwidth

Imagine a factory:

• Machines work extremely fast

• But parts arrive slowly on a conveyor belt

Adding more machines doesn’t help.
The belt is the problem.

In hardware terms:

• The machines are compute units

• The belt is memory bandwidth

Deep learning accelerators exist to fix this imbalance.

---

0.5 The Three Rules That Shape All AI Hardware

Every GPU, TPU, and accelerator follows these rules:

1. Maximize data reuse

   • Use the same numbers many times before fetching new ones

2. Move data as little as possible

   • Prefer on-chip memory over off-chip memory

3. Trade flexibility for throughput

   • Do fewer things, but do them extremely fast

Once you see these rules, GPU and TPU designs stop looking mysterious.

---

0.6 A Preview of What’s Coming

In the next chapters, we will show:

• Why matrix multiplication dominates everything

• How the Roofline model explains performance limits

• Why GPUs use thousands of threads

• Why TPUs use systolic arrays

• Why scaling models is harder than building fast chips

And most importantly:

GPUs and TPUs are not competitors —
they are different answers to the same constraints.

---

Chapter 0 Takeaway

If you remember only one thing:

Deep learning hardware exists to reduce data movement, not to increase raw compute.

Everything else is an implementation detail.

---

Next Chapter

Chapter 1: The Roofline Model — The One Graph That Explains All Accelerators

---

Before the beginning

 battle-tested way successful primers (like TPU Deep Dive, Scaling Laws, CUDA blogs) are organized, and why they work.

---

1️⃣ Start From the Workload, Not the Hardware (Most Important)

Best primers do NOT start with GPUs or TPUs.
They start with “what problem are we trying to run fast?”

Chapter 0: Why Deep Learning Is Different

• Why matrix multiply dominates

• Why data movement > compute

• Why CPUs fail at scale

This matches:

• TPU Deep Dive

• How to Scale Your Model

• NVIDIA CUDA blogs

📌 Mental anchor:

“All accelerators exist to make GEMM cheap.”

---

2️⃣ Introduce the Roofline Model Early

This is the universal unifier.

Chapter 1: The Roofline Model (Plain English)

• FLOPs vs memory bandwidth

• Why faster ALUs don’t help

• Why SRAM is gold

Once readers understand this:

• GPUs make sense

• TPUs make sense

• HBM suddenly matters

📌 Almost every good hardware DL talk implicitly assumes roofline thinking.

---

3️⃣ One Canonical Operation: Matrix Multiply

Do NOT explain CNNs, RNNs, Transformers separately at first.

Chapter 2: Everything Is MatMul

• Convolution → GEMM

• Attention → GEMM

• MLP → GEMM

Explain:

• Tiling

• Data reuse

• Blocking

📌 This is how Google’s TPU paper is written.

---

4️⃣ Then Introduce Hardware as Answers to the Same Problem

Now you can show GPUs and TPUs as different solutions to the same constraints.

Chapter 3: GPU — Latency-Hiding Machines

• Thousands of threads

• SIMT

• Caches + shared memory

• Tensor Cores

Chapter 4: TPU — Dataflow Machines

• Systolic arrays

• Deterministic data movement

• Large on-chip SRAM

• Compiler-controlled scheduling

🧠 Key insight:

GPU = “hide memory latency”
TPU = “avoid memory latency”

---

5️⃣ Put Scaling After Single-Chip Understanding

Most people get scaling wrong because they start here.

Chapter 5: Scaling One Chip

• Batch size

• Arithmetic intensity

• Model parallelism vs data parallelism

Chapter 6: Scaling Many Chips

• AllReduce

• Interconnect bandwidth

• Pipeline parallelism

This is where “How to Scale Your Model” fits naturally.

---

6️⃣ Precision & Sparsity as Optimization Knobs

Only now introduce:

• FP32 → FP16 → BF16 → INT8

• Sparsity

• Quantization

📌 Explain them as:

“Ways to increase arithmetic intensity or reduce bandwidth pressure.”

---

7️⃣ One Unifying Visual Per Chapter

Great primers always have:

• One killer diagram

• One mental model

• One takeaway sentence

Example:

• GPU: “Massively threaded latency hider”

• TPU: “Clocked matrix factory”

• HBM: “Compute’s oxygen supply”

---

8️⃣ Where to Put This Primer (Practically)

Best Formats (Ranked)

1. Living Web Document (like TPU Deep Dive)

   • Easy diagrams

   • Linkable

   • Evolves with hardware

2. Blog Series

   • Each chapter = one post

   • Easy to share

3. Open PDF / GitHub Pages

   • Versioned

   • Community contributions

📌 Avoid:

• Pure academic papers (too dense)

• Video-only (hard to reference)

• Slides without narrative

---

9️⃣ Reference Exemplars (Study These)

If you want to match the gold standard, study:

• TPU Deep Dive (Google)

• NVIDIA CUDA Blog

• Scaling Laws & “How to Scale Your Model”

• Eyeriss paper

• Vivienne Sze’s MIT lectures

---

🔑 Final Recommendation (If You’re Writing This)

Structure it like this:

1. Workload → Constraints

2. Roofline

3. MatMul

4. GPU as a solution

5. TPU as a solution

6. Scaling

7. Precision & sparsity

8. What comes next

TPU

 https://chewingonchips.substack.com/p/1201-day-1 -- Chapter 7 of Computer Architecture sixth edition by Henry & Patterson.

Biggest user of TPU

How many TPUs? - 5 million TPUs to be produced in 2027 and 7 million in 2028.