Global-Scale 3D Mapping: The Data Challenge

To localize anywhere, you need maps everywhere. Building 3D maps at global scale is a data problem before it's an algorithm problem.

Data Sources

Crowd-Sourced Images

Meta has access to billions of geolocated images from Facebook, Instagram, and user-shared content. The upside is massive coverage, especially in populated areas. The downsides: variable quality, privacy constraints, and a distribution that's anything but uniform.

Dedicated Capture

Vehicles or pedestrians with calibrated camera rigs capturing specific areas. You get high quality, controlled conditions, and known accuracy, but it's expensive and doesn't scale to everywhere.

Third-Party Maps

Partnerships with mapping companies, government data, open-source maps. Professional quality and existing coverage, at the cost of licensing, update frequency you don't control, and format compatibility work.

User Contributions

AR device users contribute mapping data during normal use. That data is always fresh and covers where users actually go. The open questions are quality control, privacy, and opt-in rates.

Quality vs Coverage Trade-off

Survey-grade equipment gives you centimeter accuracy and complete coverage within the capture area, at $100+ per square meter. Crowd-sourced user photos give you meter-level accuracy before refinement and spotty coverage that depends on photo density, at roughly the marginal cost of processing. We need both: crowd-sourced for coverage, high-quality for validation.

How do you cover the long tail?

80% of the world's photos are of 1% of locations. Tourist sites have millions of photos; suburban neighborhoods have almost nothing. In July I wrote that Meta's billions of geolocated photos could bootstrap worldwide mapping. They bootstrap the 1% of the world that people photograph. The other 99% is the actual job. For VPS to be useful everywhere, we need maps everywhere, which means solving the coverage long tail. The approaches on the table: incentivize capture in under-mapped areas, lower the quality thresholds for sparse regions, and fall back to GPS + VIO where no map exists.

Privacy at Scale

Using public photos for mapping raises real questions. Bystander faces end up in images, the problem I first wrote about for a headset camera, private property is visible from public spaces, and aggregation can reveal patterns nobody intended to share. The protections we're building in: face blurring in all processing, license plate detection and removal, opt-out mechanisms for property owners, and differential privacy for location aggregates. Privacy here is a constraint on everything we build.

Processing Pipeline

Getting from raw images to a map takes six steps: filtering (quality, duplicates, inappropriate content), georegistration to align everything to a coordinate system, Structure from Motion for sparse 3D, Multi-View Stereo for dense 3D, semantic labeling so we know what things are, and map optimization for consistency and accuracy. At billions of images, every one of those steps is an infrastructure challenge.

The mapping service I sketched last month assumed millions of locations fed by crowd-sourced images. The assumption stands; the throughput behind it does not yet. Current state: processing capacity for about 10M images a day. We need 10x that.