cd ~/
Deep LearningOptimizationEmbedded SystemsNeural NetworksMagic Leap

Neural Networks on Embedded: The Optimization Journey

Deploying neural networks on power-constrained hardware - quantization, pruning, architecture search, and the gap between research and production.

Evyatar Bluzer
3 min read

A hand tracking model that achieves state-of-the-art accuracy on a GPU server means nothing if it can't run on a headset. Closing that gap, for the keypoint network at the center of last month's pipeline, is where most of our effort goes right now.

What Does the Compute Budget Look Like?

For hand tracking we get under 15ms latency end-to-end, under 200mW average power out of the perception slice I laid out in 2016, under 50MB of model size, and hardware that amounts to an ARM CPU, a DSP, and a limited GPU - the no-accelerator bet from spec freeze. A typical research model sits at 50M parameters, 10B FLOPs, and 100ms inference on a mobile GPU.

We need 100x improvement.

Four optimization levers from research model to target hardwareA research model box at the top, an arrow down into a container of four side-by-side boxes (architecture, quantization, pruning, distillation), and an arrow down to a highlighted target hardware box. Research model50M parameters, 10B FLOPs,100 ms on a mobile GPUOptimizationOn target hardware(ARM CPU, DSP, limited GPU)12 ms of the 15 mslatency budget, no marginArchitectureResNet-50 (25M params)to a custom net(800K params),similar accuracyQuantizationFP32 to INT8,quantization-aware:4x less memory,2-4x faster, ~1% lossPruning50% structuredsparsity, under 2%accuracy dropDistillationteacher's soft targets,+5-10% accuracy overtraining the smallnet alone
The four levers between a 100 ms research model and 12 ms on the target hardware, each with the gain and accuracy cost measured below; 12 ms of a 15 ms budget leaves no margin.

Optimization Strategies

Architecture Efficiency

Start with an efficient architecture. MobileNet-style depthwise separable convolutions take 8-9x fewer operations than standard convolutions, EfficientNet compound scaling balances depth, width, and resolution, and inverted residuals give you the expand-depthwise-project pattern. We've moved from ResNet-50 (25M params) to a custom architecture (800K params) with similar accuracy.

Quantization

Moving from FP32 to INT8 buys a 4x memory reduction and a 2-4x speedup on DSP/NPU, at roughly 1% accuracy loss if done carefully. There are a few ways to get there. Post-training quantization is quick but costs accuracy. Quantization-aware training puts quantization in the training loop and keeps more of it. Mixed precision holds sensitive layers at higher precision. We use quantization-aware training with INT8 weights and activations.

Pruning

Two flavors here. Structured pruning removes entire filters or channels and is hardware-friendly. Unstructured pruning removes individual weights and reaches higher sparsity, but sparse ops are less well supported. We achieve 50% structured sparsity with under 2% accuracy drop.

Knowledge Distillation

Train the small model to mimic a large one:

Loss = α × TaskLoss + (1-α) × DistillationLoss

The large "teacher" model provides softer targets that transfer more information than hard labels. It's worth 5-10% accuracy over training the small model directly.

The Deployment Stack

PyTorch Model → ONNX Export → Target Compiler →
Optimized Kernels → Runtime Inference

Each step can introduce errors or performance regressions, so every stage needs automated testing.

Lessons Learned

Design for deployment from the start; retrofitting efficiency into a research model is painful, and the constraints should inform the architecture. Profile on target hardware, because desktop GPU profiling doesn't predict DSP performance and only actual silicon tells the truth. Know where you sit on the accuracy vs latency curve, since sometimes 5% accuracy is worth 2x speed. And beware the long tail: average performance doesn't capture the worst case, and some inputs may take 3x longer.

Current status: 12ms inference on target hardware. Within budget, but no margin.

Comments