cd ~/
VIOSLAMIMUSensor FusionMagic Leap

Visual-Inertial Odometry: Fusing Cameras and IMU

Deep dive into VIO algorithms - how we combine visual features with inertial measurements for robust 6DoF tracking.

Evyatar Bluzer
3 min read

Pure visual odometry fails in exactly the conditions where users need AR most: fast head motion, poor lighting, textureless environments. Fusing the cameras with an IMU is how we get around that; the SLAM primer gave the one-paragraph version, and this post is the rest of it.

Why fuse an IMU at all?

The IMU runs at 500-1000Hz against the camera's 30-60Hz, filling the gaps between frames. It measures acceleration and angular velocity directly, which gives the estimator a strong prior on motion. It also solves the scale problem, since monocular visual SLAM is scale-ambiguous while the accelerometer provides absolute scale. And it keeps working in darkness, through motion blur, and in featureless scenes where the cameras have nothing to grab onto.

The State Vector

VIO estimates a state vector including:

x = [p, v, q, ba, bg]

p  - position (3)
v  - velocity (3)
q  - orientation quaternion (4)
ba - accelerometer bias (3)
bg - gyroscope bias (3)

That's 16 parameters, but the unit-quaternion constraint on orientation takes one away, leaving 15 degrees of freedom.

IMU Preintegration

The naive approach is to integrate the IMU between camera frames and use the result as a motion constraint. The problem is that the integration depends on the current bias estimate, so every time the optimizer updates the bias you have to re-integrate everything. Preintegration folds the IMU measurements into a delta that is independent of the initial state:

Δp = ∫∫ R(t)(a(t) - ba)dt²
Δv = ∫ R(t)(a(t) - ba)dt
Δq = ∫ ω(t) - bg dt

These preintegrated deltas become constraints in the optimization, with Jacobians for updating them when the bias estimate changes.

IMU preintegration between camera framesTwo rows on a shared time axis: three tall camera-frame ticks at 0, 33 and 67 milliseconds above a comb of one-millisecond IMU ticks; a highlighted bar spans the IMU samples between the first two frames.03367milliseconds, drawn at 30 Hz and 1000 HzCamera frames, 30-60 HzIMU samples, 500-1000 Hzpreintegrated into one delta: Δp, Δv, Δqone constraint per frame pair
Camera frames against the IMU comb between them, drawn at 30 Hz and 1000 Hz from the ranges above; every sample inside one frame interval folds into a single preintegrated delta that the optimizer treats as one constraint.

Tightly-Coupled vs Loosely-Coupled

In a loosely-coupled system, visual odometry runs independently and gets fused with the IMU in a separate filter. That's simpler to implement, but information is lost in the VO abstraction, so the result is suboptimal. A tightly-coupled system jointly optimizes the raw visual measurements and the IMU, which buys better accuracy at the cost of complexity and compute we're short on. We're implementing tightly-coupled; for AR, the accuracy gain is worth the complexity. This is the tight VIO core from the fusion architecture I laid out in May, now being built.

Initialization

VIO needs good initial estimates to converge, and there are two ways to get them. Static initialization holds the device stationary and estimates the gravity direction and gyro bias. Dynamic initialization jointly estimates motion, gravity, scale, and biases from a short motion sequence. Static is easier but requires user cooperation, and nobody wants to hold their headset still before it starts tracking, so we need robust dynamic initialization for an instant-on experience.

Failure Modes

VIO fails when:

  • The IMU saturates (acceleration > 16g, rotation > 2000°/s)
  • Visual deprivation runs long (>1s without features)
  • Bias changes rapidly (temperature transient)

Detecting and recovering from these failures matters as much as steady-state accuracy. Next month I'll cover the mapping side of SLAM.

Comments