cd ~/
QuestVPSIntegrationEmbeddedMeta

Integrating VPS with Quest: On-Device Constraints

The challenges of bringing visual positioning to Quest headsets - power, latency, and the cloud-device split.

Evyatar Bluzer
3 min read

VPS has to run on Quest, which means mobile hardware with strict constraints. Familiar territory from my Magic Leap days.

Quest Hardware Reality

The processor is a Snapdragon XR2, an ARM-based mobile SoC, with 6GB of memory shared between the system and apps. Battery life is precious - every mW matters - and the thermal envelope is an enclosed headset with limited cooling.

VPS competes for that budget with tracking (VIO is already running), rendering at 60-90Hz, the user's actual application, and Guardian, the safety boundary. We get scraps of what remains.

The On-Device vs Cloud Split

Real-time tracking must stay on-device because it can't tolerate network latency. Feature extraction stays on-device for privacy, so no raw images are ever sent. Pose refinement stays local too, since it updates continuously. The cloud gets what can afford a round trip: initial localization once per session, map retrieval against the large database, and map updates via background sync. This is the split I drew in August before anyone had run it on an XR2, and it survived contact with the hardware unchanged.

The Latency Budget

For initial localization:

Image capture:        0ms (start)
Feature extraction:  50ms (on-device)
Network upload:     100ms (features, not images)
Cloud retrieval:    150ms (find relevant map)
Cloud matching:     100ms (compute pose)
Network download:    50ms (pose result)
On-device verify:    50ms
Total:             500ms

October's budget had no line for the network. Putting 150ms of round trip in meant retrieval grew from 100ms to 150ms and matching plus pose shrank from 200ms to 100ms to keep the total under the same 500ms. 500ms is acceptable for an initial fix. But it means users wait half a second when they start an experience.

October and January latency budgetsTwo stacked horizontal bars, one above the other, divided into labeled segments, with two segments of the lower bar highlighted and a dashed line at 500.501001505050501501005010050FeaturesRetrievalMatchingPoseVerifyFeaturesCloud retrievalCloud matchingVerifyUploadDownloadOctober 2020January 2021no network line500ms end-to-end, both budgets
October's and January's budgets for one initial localization, segment by segment in milliseconds: the two network legs are new, retrieval grew to 150ms, and matching plus pose shrank to 100ms to hold the same 500ms line.

What happens after the first fix?

After initial localization we track continuously with VIO. VPS provides the initial pose, VIO integrates motion from there, and periodic VPS re-queries correct the drift. Drift accumulates at ~0.1% of distance traveled, so a re-query every 10 seconds keeps the error bounded.

On-Device Optimizations

Getting feature extraction to run on Quest took the usual embedded playbook, the one I ran for V2 hand tracking: INT8 quantization for a 4x speedup with minimal accuracy loss, pruning away 50% of the model parameters, offloading to the Hexagon DSP when possible, and batching frames together when latency allows. We're at 50ms for feature extraction. The target is 30ms.

Privacy Architecture

The key principle: raw images never leave the device.

Camera → On-Device Processing → Features → Cloud → Pose
         (images stay here)      (abstract representation)

Features are designed to be non-invertible, meaning you can't reconstruct the image from them.

Testing Integration

The Quest-specific test suite covers power consumption monitoring, thermal behavior under sustained use, memory allocation tracking, and end-to-end latency measurement. Every code change gets validated on real Quest hardware.

Comments