Neural Map Compression: Shrinking 3D Maps 100x

Every conversation about scaling VPS eventually hits the same wall: map size. A city block runs around 100MB of point clouds and features, which is how the original architecture got to petabytes. Neural compression can shrink that dramatically while keeping localization accuracy close to the original. It was the first technical bet on the 2022 list, and it is what we have been working on.

Why compress maps at all?

A traditional map stores explicit 3D points plus descriptors - roughly 100MB per city block - and a query means searching through all of those points. A neural map replaces that with a learned implicit representation, about 1MB of network weights, queried by running inference. That is a 100x reduction, and it buys three concrete things: maps small enough to live on device with no cloud query, faster retrieval because the search space shrinks, and less data over the wire.

NeRF-Based Approaches

Neural Radiance Fields (NeRF) learn a scene representation:

(x, y, z, direction) → Network → (color, density)

For localization the requirements are different from rendering. Features need to stay consistent across viewpoints, be distinctive enough for matching, and pack into a compact representation. So we are adapting NeRF for localization rather than rendering, which changes what the network has to be good at.

Implementation

Training

We start from the images and poses produced by mapping and learn a network that encodes the scene:

Scene Images → Encoder → Compact Representation → Decoder → Synthesized Views

Trained to minimize reconstruction error.

Localization

Given a query image, find the pose that best explains it under the neural map:

Query + Neural Map → Optimization → Best Pose

The pose gets refined iteratively until the synthesized view matches the query.

Results

On the VPS benchmark, the traditional map gives 73% recall@1m at 100MB. The neural version gives 68% recall@1m at 1MB, and growing the network to 2MB recovers 71%. A few points of recall for two orders of magnitude in storage. Often an acceptable trade.

Hybrid Approaches

Pure neural representations lose some of the distinctive features that matching relies on. The hybrid keeps a neural representation for most of the scene and explicit features for distinctive landmarks, which gets you small and accurate at the same time. Our current best hybrid sits at 5MB and hits 72% recall@1m, against the 100MB baseline's 73%.

Challenges

None of this is free. Training time is the big one: each location needs individual training, hours of GPU time apiece. Update latency follows from that - we cannot quickly refresh a neural map when the scene changes, and scenes change constantly. And neural interpolation can fail at unusual viewpoints in ways explicit maps do not.

Deployment Path

The rollout has three stages: offline compression, where maps get compressed before deployment; hybrid serving, neural for the bulk of the scene with explicit features for disambiguation; then device-side maps, downloaded once and used without connectivity. The target for that last stage is Quest 3. Whether we can retrain neural maps fast enough when the world changes underneath them is the question that decides it.