cd ~/
ProfilingPerformanceInfrastructureEmbedded SystemsMagic Leap

Building Profiling Infrastructure for Embedded Perception

Creating tools and processes for systematic performance measurement on embedded hardware - the foundation of optimization.

Evyatar Bluzer
2 min read

You can't act on "we need to optimize the SLAM pipeline." You can act on "the feature matching stage consumes 340mW and takes 8.2ms on 640x480 input." Getting from the first sentence to the second requires infrastructure, and that is what this month went into.

What Do We Actually Measure?

Four things. Timing: per-function execution time, end-to-end latency, and jitter with its worst-case outliers. Power: per-subsystem consumption, power over time rather than the average alone, and correlation with algorithmic phases. Memory: peak allocation, bandwidth utilization, and cache hit rates. Thermal: junction temperatures, skin temperature at key points, and thermal throttling events.

The Profiling Stack

Hardware Layer

Power monitors on each rail (INA226 or similar), thermal sensors both on-chip and external, and a high-speed DAQ for synchronized capture.

Firmware Layer

Hardware performance counters, timestamping infrastructure, and a trace buffer with minimal overhead.

Software Layer

Instrumentation macros that can be toggled at compile time, statistical aggregation, and automated regression detection.

Visualization

Timeline views showing function execution, with power overlaid on the same timeline, plus thermal heatmaps over time.

The profiling stackFour wide boxes stacked bottom to top, hardware, firmware, software and visualization, each listing its components, joined by upward arrows labeled with what each layer passes up, ending in a highlighted box at the top quoting a per-stage power and timing figure. "The feature matching stage consumes 340 mW and takes 8.2 ms on 640x480 input" Visualization Software layer Firmware layer Hardware layer timeline views of function execution with power overlaid on the same timeline · thermal heatmaps over time instrumentation macros toggled at compile time · statistical aggregation · automated regression detection hardware performance counters · timestamping infrastructure · trace buffer with minimal overhead power monitors on each rail (INA226) · on-chip and external thermal sensors · high-speed DAQ for synchronized capture rail power and temperatures, captured in sync counters, timestamps, trace records per-function timing, power, memory, thermal a number per pipeline stage
The four-layer profiling stack from the bottom up, with what each layer hands to the one above; what comes out of the top is the per-stage number the opening sentence asks for.

Early Findings

The infrastructure is already paying off. Memory bandwidth, it turns out, is the bottleneck - CPU cycles are cheap, moving data is expensive, and we're memory-bound, not compute-bound. In July I ranked feature detectors by milliseconds per frame and treated feasibility as a compute question. I had the wrong bottleneck. Power scales super-linearly with clock: running at 80% clock uses ~60% power, so it's often better to run slower. Thermal varies by use case - portrait mode (device vertical) has 40% worse cooling than landscape due to convection patterns. And jitter matters for VIO: even when average latency is good, occasional 50ms spikes cause tracking loss, which fits how little timing slack the fusion stack has.

Process Changes

We've instituted power and latency regression tests on every commit, mandatory profiling data in code reviews for critical paths, and weekly "perf review" meetings. This is cultural change as much as technical change, and the cultural half moves slower.

Comments