6DoF Localization: From Image to Pose

Every VPS request reduces to one question: given an image from a device, where is that device in the world?

The Localization Pipeline

Query Image → Feature Extraction → Image Retrieval →
Feature Matching → Pose Estimation → Verification → 6DoF Pose

Each stage has multiple viable approaches. This is the set we landed on.

Feature Extraction

The job is converting an image into a compact representation for matching. Classical approaches like SIFT and ORB give you hand-crafted descriptors with well-understood behavior, but limited robustness to viewpoint and lighting change. Learned approaches - SuperPoint, D2-Net, R2D2 - use a neural network to detect and describe keypoints and hold up better under condition changes, at the price of needing training data. We use learned features for robustness, a route I was only exploring for low light at Magic Leap, with a classical fallback for edge cases.

Image Retrieval

Next we find which part of the map database is relevant at all. Global descriptors compress the entire image into a single vector (NetVLAD, GeM), and the query becomes: which database images are most similar to this one? At world scale that means nearest-neighbor search over billions of database images, so we lean on hierarchical coarse-to-fine search, learned hash codes for approximate NN, and geographic pre-filtering using device GPS. The target is top-100 candidates in under 100ms.

Feature Matching

Query features get matched to 3D map features. For each query keypoint we find candidate matches in the retrieved map regions, use descriptor distance for the initial correspondence, and apply Lowe's ratio test to reject ambiguous matches. The failure cases are predictable: false matches where similar descriptors sit at the wrong location, repeated structures (many buildings look alike), and viewpoint differences, where the query view can differ significantly from the map.

Pose Estimation

Given 2D-3D correspondences, compute the camera pose. PnP (Perspective-n-Point) is the classic approach:

2D image points + 3D world points → 6DoF pose

With outliers in the mix it becomes RANSAC + PnP: sample minimal sets of correspondences, compute a pose from each sample, score by inlier count, then refine using all inliers. Typically you need 50-100 inliers for a robust pose.

How do you know the pose is right?

Not every pose estimate is correct, so we verify before trusting one. We check the inlier ratio (percentage of matches consistent with the pose), geometric consistency via the reprojection error distribution, temporal consistency (is this pose plausible given the previous poses), and semantic consistency, meaning the scene content matches what the map expects. The resulting confidence score decides whether to use the pose or fall back to GPS.

Latency Budget

The end-to-end target is under 500ms from image capture to pose.

On-device optimization matters most for feature extraction, cloud optimization for retrieval, which is the device-cloud split from August expressed in milliseconds.

Failure Modes

VPS has to fail gracefully. No map coverage, the long tail from last month, means falling back to GPS. Low confidence gets surfaced as uncertainty rather than a wrong answer. Repeated structures trigger a request for more images, and a changed scene gets flagged as a possibly stale map. Users should trust VPS when it's confident, and know when it's not.