cd ~/
System ArchitecturePerceptionMixed RealityHardwareMagic Leap

Perception System Architecture: Putting It All Together

Architecting a complete perception system for mixed reality - from sensor selection to software pipeline to system integration.

Evyatar Bluzer
3 min read

After months on individual components - the depth sensor trade, SLAM, sensor fusion - this post steps back and lays out the complete perception system.

System Requirements

We need 6DoF head tracking at under 1mm position and under 0.1° orientation accuracy, plus room-scale environment mapping - a 3D mesh at ~1cm resolution - and plane detection for horizontal and vertical surfaces so content has somewhere to sit. On top of that: hand tracking with a 25-joint skeleton at 30Hz (a stretch goal) and eye tracking that yields a gaze vector for foveated rendering and interaction. All of it within a 1.5W perception budget, which lands inside the 1-2W I guessed in July, and under 20ms latency.

Sensor Suite

After extensive prototyping:

SensorPurposeResolutionRate
Tracking cameras (x2)VIO, SLAM640x48060Hz
Depth cameraMeshing, plane detection320x24030Hz
Eye cameras (x2)Eye tracking320x32090Hz
IMUHigh-rate motion-1kHz

The two tracking cameras provide stereo and wide coverage. The depth camera supplements them with metric depth for meshing. Eye cameras run faster for responsive gaze tracking. This is leaner than the suite I listed in May: the magnetometer is gone, and the dedicated hand-tracking cameras didn't survive either, so the 25-joint stretch goal has to come out of the sensors already in the table.

Processing Architecture

                    ┌─────────────────────┐
                    │      DSP Core       │
                    │  - Feature extract  │
                    │  - Depth filtering  │
                    └──────────┬──────────┘
                               │
┌──────────┐         ┌────────▼────────┐         ┌──────────┐
│  Sensors │────────►│    CPU Cores    │────────►│ Display  │
│          │         │  - VIO/SLAM     │         │ Pipeline │
└──────────┘         │  - Fusion       │         └──────────┘
                     │  - Eye tracking │
                     └────────┬────────┘
                              │
                    ┌─────────▼─────────┐
                    │    GPU/NPU        │
                    │  - Meshing        │
                    │  - Hand tracking  │
                    │  - ML inference   │
                    └───────────────────┘

Data Flow

Critical path (head tracking):

  1. Camera frame captured (t=0)
  2. IMU propagation for immediate pose (t=1ms)
  3. Feature extraction on DSP (t=5ms)
  4. VIO update on CPU (t=10ms)
  5. Pose delivered to display (t=12ms)

Total latency: 12ms, leaving 8ms of buffer for the display pipeline.

Head-tracking critical path, 0 to 20 msA waterfall of five rows on a 0 to 20 millisecond axis: a point at zero for frame capture, then four dark bars stepping right to 1, 5, 10 and 12 milliseconds, a highlighted vertical line at 12 milliseconds, and a light bar from 12 to 20 labeled 8 ms of buffer.0 ms5 ms10 ms15 ms20 msmilliseconds after frame capture; 20 ms is the motion-to-photon limitCamera frame capturedIMU propagation, immediate poseFeature extraction on the DSPVIO update on the CPUPose delivered to the displayDisplay pipelinet = 01 ms5 ms10 ms12 ms8 ms of bufferpose delivered at 12 ms
The head-tracking critical path as a waterfall of the timestamps listed above: four stages reach a delivered pose at 12 ms, leaving 8 ms of the 20 ms motion-to-photon limit for the display pipeline.

Interface Contracts

Between subsystems we're defining clear APIs: a pose service that provides head pose at any timestamp (interpolated or extrapolated), a map service for spatial anchors, meshes, and plane primitives, and a gaze service that provides eye gaze rays for rendering and input. Each service carries defined latency, accuracy, and failure mode contracts.

What's Still Unsettled?

Depth vs stereo for meshing: depth is better, but costs more power. Eye tracking accuracy requirements, which depend on the display architecture. And persistent maps: how much storage, and what the privacy implications are.

December will be spec freeze. Time to commit.

Comments