Andrei Roibu, PhD

Applied AI Systems Engineer with Oxford PhD

2 minute read

The transformative power of LLMs is undeniable, but the real challenge is bringing them to resource-constrained edge devices.

Smart Glasses Cannot Exist Without LLMs

Why would anyone buy smart glasses when they already have a smartphone that does it all?. The answer is the promise of a seamless, hands-free, private AI assistant, packaged in a sleek, lightweight, and stylish design.

The core challenge is that large language models (LLMs) consist of billions of parameters, requiring not only a large memory footprint, but also significant computational resources to run (see Section 2.2.3 of my PhD Thesis). By design, edge devices like smart glasses are resource- and area-constrained, with limited processing power, memory, and battery life. This situation opens a resource gap, forcing large AI models to the cloud, creating a series of challenges, from high latency to privacy concerns.

The Solution is as Simple as 1, 2, 3

Bridging this gap is a story in three parts.

1. We Start by Making Models Smaller: Compression & Optimisation

First, we shrink the model without compromising its performance. Techniques such as quantisation (using lower-precision numbers), pruning (removing redundant weights), and knowledge distillation (training a smaller model to mimic a larger one) can drastically reduce an LLM’s size. For instance, methods like eDKM clustering have already compressed LLaMA models from over 12 GB to a more manageable 2.5 GB with minimal impact on performance.

2. We Then Make Them Smarter: Efficient Inference

Second, we make the model run more efficiently. The biggest bottleneck in modern computing is not the computation, but the slow and energy-intensive act of moving data between memory and the processor, which can be up to 100x slower and more power-hungry. Several key innovations can help with this problem:

  • FlashAttention: A re-engineered attention mechanism that uses operation tiling to reduce the number of memory accesses;
  • KV Caching: A technique that reuses Key/Value matrices computed previously, by caching them in memory and reusing them.
  • Grouped-Query Attention (GQA): An upgrade to KV Caching, which reduces the number of Key-Value pairs we need to store in memory by grouping several heads to share a single Key-Value pair, leading to reduced data movements and memory usage.

3. Finally, We Upgrade the Hardware: Next-Generation Edge Devices

Finally, the hardware itself is evolving. We are seeing a new generation of chips emerging, such as the Qualcomm AR1 Gen 1 and the Apple Neural Engine, explicitly designed to run complex AI models at the edge. Designers often pair these chips with High-Bandwidth Memory technologies like LPDDR5X, designed for fast data access on power-constrained devices.

Yet, even these do not solve the fundamental data movement bottleneck. Addressing this situation is where novel concepts like In-Memory Computing (IMC) come into play, which perform computations directly in the memory. You can read more about this in a case study on IMC in LLM Accelerators.

Truly Personal AI Assistants Require a Multidisciplinary Approach

Deploying LLMs on edge devices is a multidisciplinary challenge that demands innovation across the entire stack, from model compression to novel hardware. The industry is closing the gap by advancing in these three areas, enabling the dream of running powerful, private AI assistants on resource-constrained devices like smart glasses, seamlessly integrating them into our daily lives.