Edge AI: considerations when moving models beyond the data centre

Most people meet AI through a chat window, with the model running on a rack of GPUs in a data centre far away. But a growing share of AI runs somewhere else: on a phone, in a camera, inside a robot, or on a small server in the building where the data is produced. That is what people mean by edge AI. Moving a model there changes almost everything about how you build it.

Why leave the data centre?

• Latency. A round trip to a distant server takes tens to hundreds of milliseconds, and much more on a poor connection. A robot avoiding an obstacle, or a production line rejecting a faulty part, cannot wait for that.
• Privacy and data rules. Video from a factory floor, medical images or customer documents may not be allowed to leave the premises. Processing them where they are created avoids the question.
• Connectivity. Fields, ships, mines and warehouses do not always have a reliable link. A model that runs locally keeps working when the link drops.
• Cost at scale. Sending every frame of every camera to the cloud is expensive in bandwidth and compute. Often the edge filters, and only what matters is sent on.

Hardware constraints

Edge hardware is limited in four ways at once:

• Memory. A model's weights must fit, together with everything else the device runs.
• Compute. Far less than a data-centre GPU, so large models run slowly or not at all.
• Power. A battery-powered device has a budget measured in watts, sometimes less.
• Heat. A sealed box in the sun throttles its own processor when it gets hot, so the speed you measure on your desk may not be the speed you get in the field.

Adapting the model to the hardware

Quantization stores weights (and often activations) with fewer bits. Training usually happens in 32-bit or 16-bit floating point; storing weights as 8-bit integers makes them about four times smaller than 32-bit floats, and 4-bit formats about eight times smaller. Smaller weights mean less memory traffic, which is often the real bottleneck, and integer maths is cheaper on many chips. The cost is some accuracy, which ranges from negligible to serious depending on the model and the method, so it has to be measured on your own task.

Distillation trains a small "student" model to imitate a large "teacher". The student will not match the teacher everywhere, but on a narrow task it can come close at a fraction of the size.

Pruning removes weights or whole structures that contribute little. It helps most when the hardware and runtime can actually skip the removed parts.

Choosing a smaller architecture is often the biggest win of all: a model designed for mobile or embedded use from the start usually beats a large model squeezed down after the fact.

Runtimes and hardware

The same model can run several times faster or slower depending on how it is executed. Runtimes such as ONNX Runtime, TensorRT for NVIDIA hardware, TensorFlow Lite (now LiteRT) for mobile and microcontrollers, and llama.cpp for running language models on ordinary CPUs and GPUs each optimise for different hardware. Many devices also include an NPU, a processor built specifically for neural-network maths, which only helps if your runtime and model actually use it.

Measure what matters

Benchmarks on a spec sheet rarely survive contact with a real device. Before deciding, measure on the hardware you will ship:

• Latency percentiles, not averages. The 95th percentile tells you how slow the bad moments are, and the bad moments are what a control loop or a user notices.
• Throughput under load, with the rest of the system running.
• Power and temperature over time. Run it for an hour, not a minute, and see whether speed holds.
• Accuracy on your own data, after quantization and every other change.

The hybrid pattern

The edge and the cloud are not either-or. A common design runs a small, fast model on the device for the decisions that cannot wait, and sends the hard or uncertain cases to a larger model in the data centre. A camera might detect locally that a person has entered a restricted area, act immediately, and send a few frames for closer analysis afterwards. A robot keeps its perception and control loop on board, while planning and reporting can live elsewhere.

That split, deciding what has to run where, is where much of the real engineering in AI products happens. It is also the part of our work we find most interesting: making a model, its software and the machine it runs on fit together.