Write-ups

Technical writing, notes, and thoughts on systems and AI.

On Memory, Inference, and Overcapacity in AI

AI LLM inference edge-computing

Large language models and transformers are very, very powerful.

However, my claim is that a large portion of what LLM inferences need by companies and corporations, as well as end consumers in their day-to-day lives, is something that is constrained by memory bandwidth and memory capacities.123

  1. LLM Inference Performance Engineering Best Practices ↩

  2. Memory Constraints in LLM Inference ↩

  3. llama.cpp - LLM inference in C/C++ ↩