cd ~/
PrivacySecurityEthicsSpatial ComputingMagic Leap

Privacy in Spatial Computing: What the Sensors See

The privacy implications of always-on spatial sensors - what data we collect, how we protect it, and the trust we must earn.

Evyatar Bluzer
3 min read

An AR headset with cameras and depth sensors is, functionally, a surveillance device you wear on your face. The same capabilities that enable magical experiences also enable an unprecedented invasion of privacy, which is why I listed privacy as an open question at the architecture stage and why we have to get this part right.

What do the sensors actually see?

Everything. RGB cameras capture whatever is visible: faces, documents, screens, private moments. Depth cameras capture the 3D geometry of spaces - room layouts, object shapes, even body measurements. Eye cameras record gaze direction, which reveals attention, interest, and potentially cognitive state. Microphones pick up conversations, background audio, and voice patterns, and the IMU and GPS trace movement patterns and location history.

The Data We Need vs. The Data We Capture

For tracking, we need visual features like corners and edges, room geometry, and head motion. We do not need face identity, document contents, or location history. The principle that falls out of that gap: capture the minimum necessary, process locally, retain nothing unnecessary.

Privacy Architecture

Local processing comes first. All perception runs on-device, and raw sensor data never leaves the headset by default. Instead of storing images we store visual descriptors, mathematical representations that work for matching but can't reconstruct the original picture. Whatever maps or spatial data we do keep are encrypted at rest with keys derived from user authentication. Cloud features like shared spaces and backup require explicit opt-in with a clear explanation of what gets shared, and maps older than N days are automatically deleted unless the user explicitly saves them. That walks back last month's storage sketch, which had the cloud keeping the full map history and prefetching by location and calendar. Cloud is now opt-in, and history expires by default.

Sensor data custodyA sequence diagram with four lifelines left to right: sensors, headset, encrypted local store, cloud; three arrows carry data rightward and notes on the lifelines say what happens at each stop, with the descriptor note on the headset highlighted. SensorsHeadset, on-deviceEncrypted local storeCloud, opt-in only raw sensor data(never leaves by default) descriptors and maps(keys from user auth) shared spaces, backup(explicit opt-in only)Cameras, depth, eye, mic, IMUsee everything: faces, documents,screens, gaze, conversationsAll perception runs here:features, room geometry,head motionImages replaced by descriptorsthat cannot reconstructthe picture; faces blurredMaps older than N daysdeleted unless saved
Where sensor data can and cannot go: raw frames stay on the headset, what gets stored is descriptors that cannot rebuild the picture, storage is encrypted and expires after N days, and the cloud sees nothing without an explicit opt-in.

Adversarial Considerations

Good intentions are not a threat model, so we walk through the adversaries explicitly. If someone steals the headset, what can they access? Encryption tied to a biometric or PIN. Can a malicious app exfiltrate camera data? A strict permission model, with no raw camera access for third parties. What can we hand over to authorities? We can't provide what we don't have; local-first processing limits that exposure. And internal misuse - employees browsing user data - is countered with access controls, audit logging, and separation of duties.

The Bystander Problem

The hardest issue is the people near the device who never consented to being sensed. Every mitigation we've looked at is partial. Indicator lights showing when cameras are active can be covered. Audio announcements are annoying and easily defeated. Face detection with blurring in stored data costs compute and isn't foolproof, and social norms and etiquette are unenforceable. We're implementing indicator lights and processing-side face blurring anyway, knowing they're partial measures.

Building Trust

All of it comes down to earning and keeping trust: transparent documentation of data practices, third-party audits, quick response when issues are discovered, and giving users control and visibility into their own data. The technology is powerful enough that we don't get to be casual about any of this. Whether it's enough to earn trust for a camera you wear on your face - we'll find out.

Comments