Blog

Ten years since I entered spatial computing. Reflecting on how the field evolved and where it's going.

A full year since leaving Meta. Reflections on independence, what I built, what I learned, and what's next.

How AI coding tools changed my development workflow - from skeptic to convert in the age of Claude Code and Cursor.

Finding the right balance between consulting work and building my own products - how each informs the other.

What I've learned from building products where AI is the core, not a feature - different principles, different challenges.

Exploring multi-agent AI systems - why they matter, how they work, and what I'm building.

The first month after leaving Meta - the quiet, the projects, and figuring out what independent work looks like.

My last day at Meta. Reflections on the journey, gratitude for the team, and anticipation for what's next.

After four years building VPS, I've decided to leave Meta. The reasoning, the emotions, and what comes next.

How large language models and generative AI are changing spatial computing - from scene understanding to content creation.

Three and a half years building VPS at Meta - from inception to platform. Considering what's next.

Quest 3 brings VPS to mainstream consumers - the largest deployment yet and what we learned.

Bringing visual positioning to consumer smart glasses - extreme power constraints and new use cases.

Quest 3's improved hardware creates opportunities for VPS - better cameras, more compute, and MR-first design.

VPS evolved from concept to platform in two years - reflecting on the journey, the team, and what's next.

Launching VPS as part of Quest Pro's mixed reality system - integration challenges, launch metrics, and lessons.

Enabling users to contribute to mapping - the technical architecture, privacy model, and quality challenges.

How Quest Pro's mixed reality capabilities change the requirements for visual positioning - higher accuracy, lower latency, always-on.

Using neural representations to dramatically compress 3D maps - enabling device-side map storage and faster queries.

Reflecting on a year of building Visual Positioning Service - shipped product, scaled team, lessons learned.

Growing a team 3x in one year - hiring, onboarding, structure, and maintaining culture through rapid growth.

What it takes to run ML in production at scale - monitoring, versioning, deployment, and all the things that aren't training.

Why indoor localization is fundamentally harder than outdoor - GPS absence, feature similarity, and the path forward.

Adapting visual positioning for smart glasses form factor - different sensors, different constraints, different use cases.

From 50 to 500 mapped locations - the operational challenges of scaling 3D mapping worldwide.

Launching the first version of Visual Positioning Service to external developers - what worked, what didn't, and what we learned.

How to localize without compromising privacy - on-device processing, feature design, and differential privacy approaches.

Using high-fidelity simulation to design and validate perception hardware before silicon exists - extending synthetic data to hardware design.

Maps go stale as the world changes. Strategies for detecting and handling changes in visual positioning.

Moving from hand-crafted to learned feature descriptors for VPS - training, deployment, and performance gains.

The challenges of bringing visual positioning to Quest headsets - power, latency, and the cloud-device split.

Half a year in - what I've learned about operating at Meta scale and building VPS from inception.

Bringing synthetic data practices from Magic Leap to Meta - what scales, what doesn't, and new opportunities.

The technical pipeline for localizing a device using visual features - retrieval, matching, and pose estimation.

Building 3D maps of the entire world - data sources, quality challenges, and the path to coverage.

High-level architecture of the Visual Positioning Service - how XR devices can localize themselves in the world.

First impressions joining Meta's Reality Labs to lead Visual Positioning Service - different scale, familiar problems.

How do you transfer years of context when leaving a role? Documenting, training, and ensuring continuity.

After four years at Magic Leap, I've decided to move on. Reflections on the decision process and what made it time.

With lab access limited, simulation becomes even more critical - accelerating our investment in virtual engineering.

When the pandemic hit, our hardware development process had to adapt overnight. What we learned about remote hardware work.

What changes when AR targets enterprise instead of consumer - reliability, integration, and the perception implications.

Systematic testing infrastructure for perception systems - from unit tests to integration tests to field validation.

Reflecting on nearly four years of building mixed reality systems - what I learned, what changed, and what's next.

Techniques for transferring knowledge from synthetic to real data, and from one perception task to another.

Evolving our sensor fusion architecture for V2 - tighter coupling, failure prediction, and graceful degradation.

Enabling multiple devices to share spatial maps - the technical architecture for multi-user AR experiences.

Advanced techniques for neural network optimization on embedded systems - beyond basic quantization and pruning.

Systematic evaluation of depth sensors for V2 - from datasheets to real-world performance characterization.

Lessons from managing perception development across continents - timezone challenges, communication patterns, and building trust remotely.

Building infrastructure to generate millions of training images - the engineering of a synthetic data factory.

Advances in spatial mapping for V2 - dense reconstruction, mesh quality, and the compute challenge of real-time 3D.

Designing a full-featured hand tracking system for V2 - gesture recognition, extended range, and the training data challenge.

Techniques for maintaining visual SLAM in challenging lighting conditions - from sensor improvements to algorithmic adaptations.

Translating field feedback and technical learnings into requirements for the next-generation device.

Reflections on the year we shipped - what the crucible of a product launch teaches that no amount of planning can.

Influencing next-generation compute chip architecture for perception - what hardware do SLAM, depth processing, and ML inference actually need?

After shipping under pressure, it's time to address the technical debt we accumulated - deciding what to fix, what to rewrite, and what to accept.

What we're learning from real users using real devices - the telemetry infrastructure and insights that drive product improvement.

We shipped a mixed reality headset. Reflections on what it means, what we got right, what we got wrong, and what comes next.

The final push before device launch - triaging bugs, making hard cuts, and keeping the team functional under extreme pressure.

Using diffractive optical elements to enable eye tracking in waveguide-based AR displays - turning a constraint into an opportunity.

Using laser speckle patterns for high-precision eye tracking - the physics, implementation, and path to a patent.

The journey from a working sensor prototype to thousands of units rolling off a production line - yield, calibration, and quality at scale.

Using extreme variation in synthetic data to bridge the sim-to-real gap - theory, practice, and hard-won lessons.

Deploying neural networks on power-constrained hardware - quantization, pruning, architecture search, and the gap between research and production.

The technical pipeline for real-time hand tracking - depth sensing, segmentation, keypoint detection, and skeleton fitting.

Reflecting on my first full year building mixed reality systems - what worked, what didn't, and what I've learned.

How to generate millions of diverse, realistic environments procedurally - the key to scaling synthetic data.

Why rendering for synthetic data differs from gaming or film - physically accurate sensor simulation over visual beauty.

Why I pushed to create a dedicated synthetic data team, and the case for simulation-first perception development.

The optical design challenges of capturing the eye from millimeters away - illumination, imaging, and integration with the display system.

Building robust eye tracking for mixed reality - from pupil detection to gaze estimation to the unique challenges of see-through displays.

The privacy implications of always-on spatial sensors - what data we collect, how we protect it, and the trust we must earn.

Designing map persistence systems that let AR content stay in place across sessions - the foundation of spatial computing.

Deep dive into VIO algorithms - how we combine visual features with inertial measurements for robust 6DoF tracking.

The hidden complexities of calibrating Time-of-Flight depth sensors for accuracy across temperature, distance, and reflectivity.

The gap between textbook calibration and production-scale calibration of perception systems - where precision meets throughput.

Lessons from hiring and structuring a multi-disciplinary perception team spanning computer vision, optics, and embedded systems.

What I learned from our first major hardware spec freeze - the art of committing to specifications when uncertainty remains.

Architecting a complete perception system for mixed reality - from sensor selection to software pipeline to system integration.

Understanding the optical system design choices for depth sensing - from field of view to distortion to manufacturability.

The physics and engineering of infrared illumination systems for ToF and structured light depth cameras.

Creating tools and processes for systematic performance measurement on embedded hardware - the foundation of optimization.

The brutal realities of running computer vision algorithms on wearable hardware - where every milliwatt counts.

Why we need high-fidelity sensor simulation for developing perception algorithms, and the challenges of making simulated data match reality.

Designing a unified sensor fusion pipeline that combines cameras, depth sensors, and IMUs into a coherent spatial understanding system.

Understanding Simultaneous Localization and Mapping from an implementation perspective - the backbone of any spatial computing device.

A deep technical comparison of Time-of-Flight and structured light depth sensing approaches for mixed reality applications.

Reflections on joining one of the most ambitious hardware startups in history, and the technical challenges that await in building a true mixed-reality device.