Memory for AI

 Data pre-processing by CPU+LPDDR

Learning/inference with GPU+HBM - paurooteri

Implementing AI activation functions ie nonlinear functions for inference, shows why it comes at the cost of performance.

The best DRAMS for AI

Algorithms for Inference MIT course

Compared to alternative solutions, its packet-based processing reduces external memory moves by more than 75% for Llama 3.2 1B and Qwen2 1.5 B. Even in highly memory-bound use cases, Origin Evolution excels in producing 1000s of effective TFLOPS and dozens of tokens per second per mm2 of silicon.

"Running LLM inference in edge hardware is crucial because it reduces latency and eliminates security concerns associated with cloud-based implementations. However, deploying LLMs in resource-constrained systems poses challenges due to their large model sizes and significant computational requirements. Consequently, edge designs require specialized hardware that can effectively address their unique resource constraints, including power, performance, area (PPA), latency, and memory requirements. Moreover, innovative software optimizations are essential, including model compression, hardware optimization, attention optimization, and the creation of dedicated frameworks to manage computational and energy constraints at the edge."

ARCHITECT

 Architecture, form, space and order

101 Things I Learned in Architecture

AI beginners

organizing AI code 

boss like

I saw a kids t-shirt and read the first part

～のような ボス 

~ No yōna bosu

a boss like

I guessed from the picture of a dinosaur that ボ might be bo(ss). Feels good to be able to recognise a language I am learning and understand it.

soarxiv

 https://soarxiv.org/

Three Gulfs of LLM development

Messy middle of AI implementation

All in one AI

Controllable reasoning   "Llama-Nemotron models is their ability to toggle between standard chat mode and reasoning mode. This "reasoning toggle" allows users to dynamically control the level of reasoning performed during inference."

The blog option of Alphaxiv is great. It makes the research papers more accessible.

what is Neural Architecture Search?

As opposed to manual, NAS employs algorithms to explore a vast search space of possible architectures and identify those that perform best on a given task.

• Puzzle Framework: The NAS framework used is called Puzzle (Bercovich et al., 2024), which transforms large language models into hardware-efficient variants under deployment constraints. (Page 3)

• Block-wise Local Distillation: Puzzle applies block-wise local distillation to build a library of alternative transformer blocks, each trained independently to improve computational properties. (Page 3)
• Mixed-Integer Programming (MIP): Puzzle uses a MIP solver to select the most efficient block configuration under given constraints like hardware compatibility, latency, memory budget, or desired throughput. (Page 4)
• Accuracy-Efficiency Tradeoff: Puzzle supports multiple block variants per layer with different accuracy-efficiency tradeoff profiles, enabling users to target specific points on the Pareto frontier. (Page 4)
• LN-Ultra Optimization: During Puzzle's architecture search phase, LN-Ultra is constrained to achieve at least a 1.5x latency reduction over Llama 3.1-405B-Instruct. (Page 5)

In essence, NAS, through the Puzzle framework, allows the researchers to automatically find efficient model architectures within the Llama 3 structure, optimizing for metrics like latency, memory usage, and throughput while maintaining a desired level of accuracy. This is a key step in creating the Llama-Nemotron models, particularly LN-Super and LN-Ultra.                                                                                              

 

Collective bias when individual LLM agents interact

alphaxiv

 arxiv.org to alphaxiv.org and chat away with AI.

arxiv.org/pdf/2505.09343

to

https://www.alphaxiv.org/pdf/2505.09343

Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architectures

Trying to understand the picture:

What is cross-entropy loss?

"Cross-entropy loss is a loss function used in machine learning and optimization, particularly in classification problems. It quantifies the difference between two probability distributions: the predicted distribution from your model and the true distribution of the labels."

What is the difference between a shared expert vs routed expert?

In the context of Mixture of Experts (MoE) models, "shared experts" and "routed experts" refer to different ways of organizing and utilizing the expert sub-networks within the larger model. 

• Routed experts allow the model to specialize, with different experts handling different types of inputs.
• Shared experts process all tokens, potentially capturing general features or providing a baseline level of processing.

What is GQA?

GQA stands for Grouped-Query Attention. It's an attention mechanism used in transformer models, particularly designed to improve inference efficiency.

• The Problem GQA Solves: In multi-head attention, each attention head has its own separate Q, K, and V vectors. During inference, the key and value vectors from previous tokens need to be stored in a cache (KV cache) to efficiently handle multi-turn conversations. This KV cache can consume a significant amount of memory, especially for long sequences.

• How GQA Works: GQA reduces memory consumption by having multiple attention heads share a single set of Key and Value (KV) pairs. Instead of maintaining separate KV pairs for each attention head, multiple heads share one. This significantly compresses KV storage.