Learned Features for Visual Localization
Moving from hand-crafted to learned feature descriptors for VPS - training, deployment, and performance gains.
Classical features had a long run. SIFT and ORB carried visual localization for over a decade, and for most of that time nothing learned could touch them at their own game. That stopped being true, and the gap is now wide enough that we are making the switch.
Why switch now?
The failure modes of classical features are exactly the conditions a VPS meets in the wild: large viewpoint changes (more than 30 degrees of rotation), day versus night illumination, seasonal changes like snow and foliage, and weather like rain and fog. Learned features handle all of these better because they saw them during training, instead of hoping a gradient histogram generalizes.
The benchmarks back this up. On HPatches and Aachen Day-Night, SIFT sits at 45% localization success while SuperPoint + SuperGlue reaches 78%. That is not a margin you argue with. I wrote in October that learned features with a classical fallback would beat SIFT and ORB on viewpoint and lighting. That one was not a brave call, and it held.
Feature Learning Approaches
Three families are worth knowing.
Detection + Description (SuperPoint style)
One network jointly detects keypoints and computes descriptors:
Image → CNN → Keypoint Heatmap + Descriptor Map
It trains end to end and runs fast, which is why it tends to be the default for on-device work. The cost is a fixed grid output and the quantization effects that come with it.
Dense Description (D2-Net style)
Describe every pixel first, then detect keypoints from the description scores:
Image → CNN → Dense Descriptors → Keypoint Detection
No separate detection stage, so it is more flexible about where keypoints land. It is also slower and hungrier for memory, which matters on mobile.
Hierarchical (HLoc style)
A global retrieval network narrows the database to candidates, then local features compute the pose:
Image → Global Net → Candidates → Local Features → Pose
Best accuracy of the three, and the only one that handles large databases well, at the price of running multiple networks in a pipeline you now have to maintain.
Training for VPS
Off-the-shelf models are trained on academic datasets, and it shows. We need more geographic diversity, more condition diversity, and data that matches Quest-specific camera characteristics.
Our training data comes from three places: synthetic renders (full control, unlimited volume), real captures from VPS mapping (authentic but limited), and public datasets (diverse but outside our control). The recipe itself is unsurprising. Pre-train on the large public data, fine-tune on VPS-specific captures, then apply domain adaptation to close the synthetic-to-real gap.
Deployment Considerations
Learned features cost more than classical ones everywhere you can measure. Model size is 10MB+ against ORB's near-nothing. Inference runs 30-50ms on mobile where ORB takes 5ms. And the feature maps eat GPU memory that the rest of the tracking stack wants for itself.
We claw most of that back with knowledge distillation into a smaller student network, INT8 quantization with minimal accuracy loss, and TensorRT/NNAPI deployment for hardware acceleration. The current student model is 2MB and runs in 25ms on Quest. In January the extraction target was 30ms, down from 50; distillation got there with a little room to spare.
A/B Testing Plan
The rollout is deliberately boring: internal dogfooding with learned features first, then a small percentage of external users, measure localization success rate, expand if the metrics improve. Classical features stay available as a fallback if the learned path fails.
Results So Far
Internal testing shows a 40% reduction in localization failures, better performance in challenging lighting, and comparable latency after optimization. Broader rollout is planned for Q2 2021.