Visual Positioning System: Architecture Overview
High-level architecture of the Visual Positioning Service - how XR devices can localize themselves in the world.
We're building a system that lets any XR device know where it is in the world. This post is a first pass at the architecture.
The Problem
Picture the basic scenario. A user puts on a headset and opens an AR experience anchored to a specific physical location, say a sculpture in a park. The device has to recognize "I'm near the park", localize precisely ("I'm at position X,Y,Z with orientation R"), track continuously as the user moves, and cope with the sculpture not being where it was mapped. That whole chain is the Visual Positioning Service.
System Components
┌──────────────────────────────────────────────────────────────┐
│ VPS Architecture │
├──────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────┐ ┌─────────────────┐ ┌─────────────┐ │
│ │ Mapping │ │ Map Storage & │ │Localization │ │
│ │ Service │───►│ Retrieval │───►│ Service │ │
│ │ │ │ │ │ │ │
│ └─────────────┘ └─────────────────┘ └─────────────┘ │
│ │ │ │
│ │ ┌─────────────────┐ │ │
│ └──────────►│ Ground Truth │◄──────────┘ │
│ │ & Validation │ │
│ └─────────────────┘ │
│ │
└──────────────────────────────────────────────────────────────┘
Mapping Service
Converts raw sensor data (images, depth, poses) into 3D maps, using Structure from Motion (SfM) for sparse reconstruction, Multi-View Stereo (MVS) for dense reconstruction, and semantic understanding for landmark detection. Input is crowd-sourced images with metadata. Output is georeferenced 3D maps with visual features.
Map Storage & Retrieval
The global map database. It has to store maps efficiently at world scale, index them spatially for fast retrieval, version them as maps update, and enforce privacy controls - whose data it is, and where it can be accessed.
Localization Service
Matches device observations to stored maps. Image retrieval finds candidate map regions, feature matching establishes 2D-3D correspondences, pose estimation computes the 6DoF device pose, and a verification step handles confidence scoring and outlier rejection.
Ground Truth & Validation
Keeps the rest of the system honest: reference measurements from survey equipment, automated accuracy regression testing of the kind that blocked merges at Magic Leap, and a feedback loop that improves mapping over time.
What runs on-device and what runs in the cloud?
On-device you get real-time tracking (60Hz VIO), local feature extraction, and privacy preservation, since no images leave the device. The cloud handles large-scale mapping, because you can't run SfM on a headset; map storage, because the maps are far too large for the device; and initial localization against the global database.
The hybrid flow ties them together. The device captures images and extracts features locally, sends the features rather than the images to the cloud, the same sparse-map choice we made for sharing at Magic Leap, gets back a pose estimate, and refines it locally with VIO.
Scale Challenges
We want to map millions of locations, and even with crowd-sourcing that's a massive computational job. Storage compounds it: high-quality 3D maps are large, and millions of locations times gigabytes per location comes out to petabytes. Query load means millions of devices localizing simultaneously with low latency required. And the world keeps changing, so maps have to stay current. Meta's infrastructure helps, but the problems remain hard.
Privacy by Design
VPS touches sensitive data by definition: user location, images of the real world, presence at specific places. The principles we're holding to are minimizing data collection, processing locally when possible, clear consent for any cloud interaction, and no selling or sharing of location data. Most of that list is the same set of rules I wrote down in 2017, carried across a company boundary. Privacy review is a gate for every feature.
More details on individual components in coming posts.