cd ~/
Chip ArchitectureHardwarePerceptionComputingMagic Leap

Architecting Compute Silicon for Perception Workloads

Influencing next-generation compute chip architecture for perception - what hardware do SLAM, depth processing, and ML inference actually need?

Evyatar Bluzer
3 min read

What does perception actually need from silicon? We're starting to influence the compute architecture for future devices, so for the first time that question has real stakes attached.

Current Limitations

The current chip is a mobile SoC plus custom accelerators. The CPU handles control flow and fusion, the DSP runs feature extraction and filtering, the GPU runs ML inference (slowly), and custom blocks handle specific functions. The pain points are consistent across all of it: data movement between units burns power, memory bandwidth is the bottleneck, GPU inference is power-hungry for small models - the 800K-parameter hand network fits its budget with no margin left - and the fixed-function blocks lack flexibility.

Perception Workload Analysis

Breaking down where cycles go:

Perception cycle budgetFive horizontal bars sorted largest first: ML inference at 30 percent highlighted, then feature extraction 25, depth processing 20, tracking and SLAM 15, control and fusion 10, each with a short note on its compute character.ML inferenceFeature extractionDepth processingTracking / SLAMControl / fusion30%25%20%15%10%matrix ops, nonlinearparallel, local opsfiltering, interpolationsparse linear algebrasequential, branchy
Where perception cycles go on the current chip, from our workload breakdown: ML inference is the largest slice at 30%, and it is the slice that argues for a dedicated NPU.

Each one has a different optimal compute architecture.

What We Need

Efficient ML Accelerator

  • INT8 matrix multiply (95% of inference)
  • Flexible enough for various network shapes
  • Low power (10-50 TOPS/W target)
  • Low latency startup (no batch amortization)

Image Processing Unit

  • 2D convolution engine
  • Distortion correction (LUT-based)
  • Feature detection (Harris, ORB)
  • Stream processing (minimize memory round-trips)

Memory Architecture

  • High bandwidth for tensor operations
  • Low latency for sparse access (SLAM)
  • Scratchpad for intermediate results
  • DMA engines for background data movement

Compute Fabric

  • Ability to pipeline operations across units
  • Minimal CPU involvement in data flow
  • Power gating for unused units

Trade-offs in Discussion

Fixed function versus programmable is the classic one. Fixed is efficient but inflexible; programmable handles algorithm changes but wastes area and power. My recommendation is fixed for stable algorithms (distortion, simple filtering) and programmable for the ones still evolving (ML, SLAM).

On-chip versus off-chip memory splits the same way. On-chip is fast and efficient but limited, off-chip is large but power-hungry. So: a large on-chip scratchpad for inference, where the models fit, and accept off-chip for mapping, where the data doesn't.

Then there's the question of whether inference belongs on a dedicated NPU or on GPU compute. A dedicated NPU is efficient, GPU compute is flexible. At spec freeze in 2016 I bet we could skip a dedicated accelerator and run most inference on the DSP and GPU, with the caveat that neural nets becoming central would break the bet. They did: inference is now 30% of the cycle budget. I'm arguing for a dedicated NPU for inference with the GPU reserved for graphics. Don't share.

Working with the Chip Team

The chip design cycle is 2-3 years, so decisions made now determine what's possible in 2021, the same lock-in I described in 2016, this time from the inside. My part is providing workload characterization backed by cycle-accurate models, benchmarking competing architectures on our algorithms, defining KPIs that matter (TOPS/W at our model sizes, rather than raw TOPS), and reviewing architecture proposals for perception fit.

It's a different kind of engineering, influencing hardware through analysis rather than writing code, and the leverage is bigger than anything I could get from code alone.

Comments