On Memory, Inference, and Overcapacity in AI

by Shahbaz Shaik
AI LLM inference edge-computing

Large language models and transformers are very, very powerful.

However, my claim is that a large portion of what LLM inferences need by companies and corporations, as well as end consumers in their day-to-day lives, is something that is constrained by memory bandwidth and memory capacities.123

Memory, reliability, and failing reliably

It does not matter as much for me if the message that I send to ChatGPT takes 10–15 seconds more, as long as it doesn’t fail, as long as I don’t have to pay as much money, and as long as it doesn’t hallucinate. Given these constraints, it does not feel like the primary requirements of users is that of extremely fast inference.

Most users who are interacting with ChatGPT or with other large language models or the services that they’re trying to provide will only be able to process so much information at a particular point of time. Given this, it felt to me that ChatGPT on Android unreliably crashing when running in temporary memory mode or GitHub Copilot sometimes not being able to run or continue with the chat were bigger breaks than a model which hallucinates frequently.

Additionally, I never had to go ahead and fine-tune all of these or train all of these large language models from scratch. I just needed to depend on inference for user-use cases such as mine. For example, the usage of various deep research modes, starting from DeepSeek, Gemini, and ChatGPT, and later Claude, are all examples of how I personally was always willing to give up more time in order to go through a larger number of sources just to get more verifiable data.

These were bigger annoyances than just being confident in the unreliability of the LLM. I broadly know that I need to double check things, but the context of multiple chats being lost in GitHub Copilot in Visual Studio, and not being able to copy paste the same context reliably, were far bigger annoyances. After all, I still remember the shockwaves that earlier ChatGPT versions caused, and they were just as unreliable as small language models today when trying to use them day to day.

Can I run it locally?

The performance, the raw inference performance of locally running LLMs on consumer gaming GPUs like the 3070, NVIDIA 3070 8GB left me pleasantly surprised, especially when using quantized models.34 Secondly, there has been an improvement and increase in the use of CPUs or NPUs for inference. CPUs and NPUs require the usage of RAM or some unified RAM for storing of models.56 These have an obvious downside of being slower in terms of inference speed. However, the fact that Apple’s silicon can run a 48GB model even with slower inference locally gives credence that Edge AI, even with slower inference, would truly be valuable.7

It feels like memory and inference are two sides of the AI coin, and we have been increasing both of these to an extremely large extent.

Where would we need AI?

Most of the important inference that is needed by a user in their day-to-day lives that I can imagine, such as for NLP to text transcribing and then to perform certain tasks on the basis of that transcribing, all seem possible with a small locally running LLM that is built on top of some home assistant or some other sort of a layer if I wanted to, for example, automate my house using NLP.89 For example, a small locally running agent that is built on top of Home Assistant, but with Whisper and an NLP layer on top of it, already covers a large fraction of what I would consider important day-to-day inference needs.

Similarly, if I wanted to run some coding assistance, here latency would be a lot more significant, but it would allow me to run coding assistance without having to rely on an internet connection. Given this, it feels more feasible that users would be unwilling to pay for models that are running on cloud machines if the price for them is too high, if the hardware that they already have available is able to perform the same task, albeit slowly.

The complexity of fine-tuning transformers and orchestrating agents—maybe a mixture of experts or agents working in sequence as part of a workflow—may still require extremely large compute. But the end user who is using these trained flows will not always need the same.

Small models are catching up

Examples of Meta’s smaller models and their mobile models show that tool calling, fine-tuning, and distillation have greatly helped in bridging the gap. Current small language models that leverage large language models—either for data synthesis or task-specific distillation—are quite effective.1011 The assistance of LLMs to improve LLMs feels circular, like how GCC is written in C or how the Rust compiler was rewritten from OCaml into Rust. Once you have a working version, you use it to build a better version of itself.

Hybrid AI

Hybrid AI is something that will just improve latency and performance if we consider every user having a lot of compute available on their edge devices, such as their phones or whatever device they’re using to interact. The cost of transcribing it locally and then sending it to the cloud LLM and then receiving the outputs feels like a much better use.12 This is obviously being done, but there is scope for improvement in terms of what responsibilities each of these takes over and what responsibilities should not be transferred to the cloud for data sovereignty, privacy, and latency reasons.

Overcapacity

By overcapacity, I mean that all of these GPU workloads, etc., may only be needed for coding use cases, may only be needed for various complex image recognition tasks where a smaller transformer will not work, and so on. But for many of the day-to-day cases where an AI that is not hallucinating and is bounded with okay latency is good enough. For other AI flows, extremely high bandwidth, extremely large sizes of GPU VRAM and server RAM seems like an overcapacity.13

There is only so much that I can expect a single large language model to do, even if the inference speed is extremely high, I am limited by the sequential nature of any flows.14 So if the question is that of having parallel LLMs, then it feels like the question is no longer of a single large GPU, but that of multi-GPU, multi-agent orchestration.15 For this, a single large GPU would provide a boost, but I feel like that is not the limiting factor preventing greater user adoption.

  1. LLM Inference Performance Engineering Best Practices ↩

  2. Memory Constraints in LLM Inference ↩

  3. llama.cpp - LLM inference in C/C++ ↩ ↩2

  4. 4-bit Transformers with bitsandbytes ↩

  5. Apple Developer - On-device ML ↩

  6. Microsoft Phi Silica ↩

  7. Apple M3 Max can run 70B LLMs locally ↩

  8. Home Assistant Local Voice ↩

  9. OpenAI Whisper ↩

  10. Meta Llama ↩

  11. Distilling Step-by-Step ↩

  12. Introducing Apple Foundation Models ↩

  13. AI boom drives global memory chip shortage ↩

  14. Sequential Processing Limits in LLMs ↩

  15. Multi-Agent Orchestration ↩