Year One at Magic Leap: A Retrospective
Reflecting on my first full year building mixed reality systems - what worked, what didn't, and what I've learned.
A year ago I joined Magic Leap knowing almost nothing about mixed reality. Today I lead the perception architecture for a device that's actually taking shape, which seems like a fair moment to be honest about what worked and what didn't.
What Went Well
Technical Foundation
We have working systems: visual-inertial odometry running at 60Hz with under 1mm tracking error, a ToF depth sensor integrated and calibrated, an eye tracking prototype showing promising accuracy, and a synthetic data pipeline generating training data. The architecture decisions from early 2017 are proving sound.
Team Growth
We started the year with 3 engineers and now have 15 across perception disciplines. The technical hires are strong and already contributing, demo-driven development, a norm we set in January, has taken root as the culture, and cross-functional collaboration is improving.
Synthetic Data Bet
The synthetic data team is already accelerating our ML development. In September I picked eye tracking as the first domain because it was the most contained, and it was the right first pick: training eye tracking models on synthetic data is working better than I expected.
What Didn't Go Well
Schedule Optimism
We underestimated integration complexity. Individual components working doesn't mean the system works, and integration bugs consumed Q3-Q4. The lesson I'm carrying into next year: add 50% to integration estimates, minimum.
Hardware-Software Handoff
Communication gaps between the hardware side (optics, sensors) and the software side (algorithms) caused rework. Specs that seemed clear weren't. Written specs aren't enough - you need working prototypes across the boundary.
Power Budget Overrun
At spec freeze I wrote that we would use every bit of the 20% power margin. We used it and kept going: we're 30% over power budget. Every subsystem optimized locally, but the sum exceeds the system constraint, and now we're doing painful trade-offs. Power has to be tracked continuously at the system level, not per-component after the fact.
What Did I Actually Learn?
Perception is a System Problem
You can't optimize depth sensing without considering SLAM, and you can't optimize SLAM without considering display latency. Every subsystem's choices land in someone else's error budget, and a lot of this year went into learning which of those couplings matter.
Data is Strategy
The team that controls data generation controls algorithm development velocity. Synthetic data is strategic infrastructure.
Embedded is Different
Algorithms that work on desktop often can't be translated to embedded. Design for the target platform from day one.
Calibration is Half the Work
Brilliant algorithms don't matter if the calibration is wrong. Invest in calibration infrastructure early.
Looking Ahead to 2018
Four priorities: ship the developer kit and get real devices into real developers' hands, get back within the power budget, scale synthetic data generation 10x, and add hand tracking as an input modality.
It's going to be intense, but we're building something real, and that keeps me going. Thanks to everyone on the team who made this year possible.