Sensor Fusion Architecture for Spatial Perception

Individual sensors lie. Cameras are fooled by lighting changes, IMUs drift, depth sensors have holes. The art of spatial perception is fusing these imperfect sources into something reliable.

Why Is Fusion Hard?

Our headset will have multiple RGB cameras (wide and narrow FOV), a depth sensor (ToF or structured light, still undecided), an IMU with accelerometer and gyroscope, a magnetometer, and potentially eye cameras and hand tracking cameras on top of that.

Every one of these runs on its own terms. Sample rates span 1kHz for the IMU down to 30Hz for depth, with cameras at 30-60Hz in between. Latencies differ too: camera processing takes 10-30ms while the IMU is nearly instant. And each fails in its own way - depth fails in sunlight, cameras fail in darkness, and the IMU drifts always.

Fusion Approaches

Extended Kalman Filter (EKF)

The classical approach: maintain a state estimate (pose, velocity, biases) and update it with each sensor measurement.

Predict: x̂ₖ = f(x̂ₖ₋₁, uₖ)
Update: x̂ₖ = x̂ₖ + K(zₖ - h(x̂ₖ))

An EKF is well understood and computationally efficient. The price is linearization error, and it can't handle multi-modal distributions.

Factor Graph Optimization

Model the problem as a graph where nodes are states and edges are constraints from measurements, then solve via nonlinear least squares. This can incorporate any measurement type and handles loop closures naturally, but it's computationally expensive and requires careful marginalization.

Our Hybrid Approach

We're converging on a hybrid architecture, built around the VIO problem the SLAM primer only sketched: a tight VIO core using IMU + cameras with an EKF for low-latency tracking, a sliding window optimization refining recent poses, and a factor graph backend for map maintenance and loop closure.

Time Synchronization

Every sensor reports with a different latency, which means a naive fusion of IMU data with camera data compares measurements from different moments in time. There are a few ways to deal with it: hardware trigger synchronization, timestamping at the sensor level, and interpolation or extrapolation inside the fusion itself. Getting this wrong by even 10ms causes noticeable instability in AR. We're learning this the hard way.

One measurement's path through the hybrid stack: IMU samples at 1 kHz and camera frames 10-30 ms behind them are matched in time before the EKF core sees either, then poses pass to the sliding window and the factor graph. Rates and latencies are the ones listed above.

Next Steps

The next piece is a simulation environment, so we can test fusion algorithms before hardware is ready. More on that once it exists.