Datacenter as a Computer

datacenter-computing-spring-19 course

 The Datacenter as a Computer: Designing Warehouse-Scale Machines, Third Edition

https://pages.cs.wisc.edu/~shivaram/cs744-readings/

GPU computing era paper

Building Software Systems at Google and Lessons Learned pdf by Jeff Dean

Building Software Systems at Google video

Roofline model revisited

 https://puzzles.modular.com/puzzle_16/roofline.html

More hardware speciifcation from Cornell ECE6775 course Lecture 3.

roofline-an-insightful-visual-performance-model-for-multicore-architectures first apperaed in 2009 CACM journal.

GPU Concurrency

 A Primer on Memory Consistency and Cache Coherence, Second Edition

 NVIDIA’s virtual ISA is called Parallel Thread Execution (PTX).

A kernel refers to a unit of work offloaded by the CPU onto the GPU and typically consists of a large number of software threads.

Eaton Phil

 Operating Systems: Three Easy Pieces

https://eatonphil.com/bookclub.html

how to write deterministic and easily testable code with single control thread

Be very good at uncorrelated things

problem of the week

High level is the goal

System design case studies

https://posts.cgamedev.com/p/the-hidden-cost-of-software-libraries

If we take the chance that a tool (compiler, linker, batch, whatever) remains working for a particular codebase after one year as a given probability p, then the chance that build remains working after x years is p^xn, where n is the number of tools used in the build.

SIMD and matmul

 A Primer on Memory Consistency and Cache Coherence, Second Edition - GPU

https://vectrx.substack.com/p/simd-a-practical-guide

https://michalpitr.substack.com/p/optimizing-matrix-multiplication

https://www.henrikkarlsson.xyz/p/perceptive - Knowledge shield - drink from firehose

software concepts to understand linux

https://x.com/pulkit_mittal_/status/2007076913714766326 - HFT questions

Distinguishing yourself early in your career as leader

training a neural network

Doing a lot at once

 why-do-we-even-need-simd-instructions 

unlike many of the previous architectural innovations, MMX and its successors all require software updates in order to benefit from the new instructions, which takes much longer because not only do developers have to write and test new code, but it must propagate through intermediate layers (such as an operating system release cycle) before finding its way into the hands of end customers. 

Integrating FPU like a platform plugging in

Superscalar_processor

That detail is easy to miss if you only look at FLOPS charts. One of the barrier why “train FP16 then PTQ-quantize” is not the same as having native low-precision pathways that actually save compute and memory at the same time. That’s why Nvidia’s advantage isn’t merely that it “supports FP8,” but that it keeps moving the boundary of what is practically trainable/servable at lower precision without asking the ecosystem to rewrite itself each generation.

"Parts of the Transformer Engine that previously existed as “software over Tensor Cores” are being moved into hardware - this is ASIC-like behavior as it hs hardening a workload pattern into silicon." similar to  Instead, an alternative is to offload such frequent computations to a FPGA, which can perform the same computation in a fraction of the time than software would take

Microarchitecture what happens beneath

The raw power of Intelligence from Samuel albanie reflections 2025

fast inverse square root

https://martin.kleppmann.com/2025/12/08/ai-formal-verification.html

More learning

 (科学空间 / kexue.fm) is a science blog called Scientific Spaces that publishes posts on a wide range of natural science topics, especially math, astronomy, physics, machine learning, and optimization methods like AdamW, MuP, and diffusion models. It mixes research-level explanations with tutorials and math derivations aimed at practitioners and enthusiasts. Scientific Spaces

Several links to educational or resource pages: the MIT lecture PDF (6.189 Lecture on parallel design patterns for concurrent programming), a personal math site (Justin Skycak’s offerings including an Introduction to Algorithms and Machine Learning book/tutorial), and a page on Exploring Mathematics with Python from PSU. These focus on core computer science concepts and applied math instruction. The JustinMath site also includes reflections on advanced math/CS education outcomes (the Eurisko Sequence). groups.csail.mit.edu+2coe.psu.ac.th+2

The social snippet (X post) is praise of the experience of working with well-known open-source and systems programmers (like Ted Ts’o and Eric Biggers), highlighting admiration for highly productive coding skill. The arXiv paper 1703.09039 referenced is a foundational survey on efficient processing of deep neural networks (deep learning hardware/software tradeoffs), widely cited in embedded/efficient AI contexts. csc.ucdavis.edu+1

Finally, the Notion Transformer link discusses the Transformer neural network architecture — the deep learning model behind attention mechanisms in modern language and vision models (e.g., “Attention Is All You Need”). The JustinMath PDF and other resources complement this by offering algorithmic and machine learning foundations.

https://www.notion.so/Transformer-c0a6ab89e04543ffa855170cef2759fa

https://groups.csail.mit.edu/cag/ps3/lectures/6.189-lecture6-patterns1.pdf

https://naklecha.com/about

https://coe.psu.ac.th/ad/explore/

https://x.com/ciphergoth/status/2006446942453387675 "I've worked with some incredible people - you might have heard of Jaegeuk Kim or Ted Ts'o - and some ridiculously productive programmers - Eric Biggers, Jeff Sharkey and

@jackinwarsaw

come to mind as people who seemed to solve problems with code at a truly unearthly rate."

https://www.justinmath.com/books/#introduction-to-algorithms-and-machine-learning

https://www.justinmath.com/files/introduction-to-algorithms-and-machine-learning.pdf

https://www.justinmath.com/math-academys-eurisko-sequence-5-years-later/

https://arxiv.org/pdf/1703.09039