cd ~/
Embedded SystemsOptimizationPowerHardwareMagic Leap

Perception on Embedded: Power, Thermal, and Compute Constraints

The brutal realities of running computer vision algorithms on wearable hardware - where every milliwatt counts.

Evyatar Bluzer
2 min read

On a server, you optimize for accuracy. On a headset worn on someone's face, you optimize for not burning them. The milliwatt constraints that humbled me in February now have numbers attached.

The Constraint Triangle

The power budget is ~5-8W for the entire system - display, compute, sensors, wireless - and perception might get 1-2W of it, with the 500mW I budgeted for depth alone coming out of that slice. Thermal is stricter than you might expect: the device sits on your face, surface temperature above 41°C causes discomfort, and above 45°C you have a safety issue. Then latency: motion-to-photon must stay under 20ms to avoid nausea, and perception is in the critical path.

These constraints feed into each other. Running faster uses more power, which generates more heat, which throttles the processor, which increases latency.

What Does This Mean for Algorithms?

Algorithms that work beautifully on a desktop GPU may be completely impractical here. A few examples:

Feature Detection

SIFT: 200ms per frame -> Unusable ORB: 15ms per frame -> Marginal Custom binary descriptors: 3ms -> Feasible

Depth Processing

Full frame bilateral filter: 25ms -> Too slow Sparse depth + interpolation: 5ms -> Workable

SLAM Backend

Full bundle adjustment: 500ms -> Background only Sliding window: 30ms -> Near real-time

Per-frame timings against the 20 ms budgetA dot plot of seven named algorithms in three groups on a log time axis from 1 to 1000 milliseconds, each dot labeled with its time and the post's verdict, crossed by one highlighted vertical line at 20 milliseconds.1 ms10 ms100 ms1000 msper frame, log scaleFeature detectionSIFTORBCustom binary descriptorsDepth processingFull-frame bilateral filterSparse depth + interpolationSLAM backendFull bundle adjustmentSliding window3 ms, feasible25 ms, too slow5 ms, workable30 ms, near real-time200 ms, unusable15 ms, marginal500 ms, background only20 ms: the whole motion-to-photon budget
The seven per-frame timings above on a log axis against the 20 ms motion-to-photon limit. The verdicts are the post's own, and 20 ms is the whole budget, of which perception gets only a part.

Hardware Acceleration

We can't solve this with algorithms alone. A DSP handles fixed-function pipelines well (filtering, feature detection). A GPU handles parallel workloads well (stereo matching, neural networks). Custom silicon gives the best efficiency but takes the longest to develop. The architecture decision we make now will determine what's possible for the next 5 years.

The Profiling Discipline

Every engineer on the perception team needs to internalize:

  1. Profile before optimizing
  2. Measure power, not time alone
  3. Test on target hardware, not desktop
  4. Consider thermal over sustained workloads, not peak alone

We're building profiling infrastructure to make those four habits cheap. That work is next month's topic.

Comments