Founding the Synthetic Data Team

After a year of building perception systems, one pattern keeps showing up: we're bottlenecked on data. Today I made the case for a dedicated synthetic data team, and it got approved. I first made the case for a simulation team in June of last year, mostly as an iteration-speed argument for testing algorithms before hardware existed. It took fifteen months and a data bottleneck to get it funded.

The Data Problem

Machine learning for perception needs scale, meaning millions of labeled examples. It needs diversity across every lighting condition, environment, and user variation. It needs accuracy - pixel-perfect labels for segmentation, millimeter-perfect depth - and it needs the edge cases, the rare scenarios where systems fail.

Real data collection fails on all counts. It's expensive and slow. Diversity is limited to what you can physically capture. Human annotation has errors, so the labels are noisy. And edge cases are, by definition, rare.

The Synthetic Data Promise

In simulation you can generate unlimited data programmatically, with perfect ground truth by construction, full control over conditions, and easy synthesis of rare events. The catch: synthetic data must be "real enough" to transfer to actual sensors.

Team Charter

The team's charter covers four things: rendering infrastructure for sensor-accurate simulation, asset pipelines for scalable environment generation, domain adaptation techniques to close the reality gap, and validation protocols to prove the transfer actually happens.

Initial Focus Areas

We're starting with eye tracking - eye images in simulation, with controllable eyelid positions, pupil sizes, and gaze directions - because it's a relatively contained domain and because the learned half of the gaze pipeline needs labeled eye images at a scale real capture won't deliver. Depth sensors come next: simulating ToF and structured light, including the sensor noise models, which is also where the multi-path correction we wanted to train on simulated data has to come from. RGB features are the hardest of the three, photorealistic rendering at scale.

Hiring Profile

This profile looks different from a typical CV engineer. It's graphics expertise in real-time rendering, PBR, and ray tracing; procedural generation, creating variation programmatically; enough domain knowledge to understand what perception algorithms need; and a tooling mindset, because these are systems other people have to use. We found our first hire already, a graphics engineer from gaming who's excited about the applied ML angle.

How do we know it's working?

Three measurements: the gap (performance on a real test set after training on synthetic versus real data), marginal value (does adding more synthetic data keep improving performance?), and coverage (are we reaching scenarios impossible to capture in real data?).

This is a bet, and the alternative - scaling real data collection indefinitely - is not sustainable. The goal I've committed to: synthetic data enables better models than pure real-data training by the end of next year.