Synthetic Data at Meta Scale
Bringing synthetic data practices from Magic Leap to Meta - what scales, what doesn't, and new opportunities.
At Magic Leap I founded the synthetic data team. At Meta I'm making the case for similar investment at much larger scale.
What's Different at Meta
Compute, first: access to datacenter-scale GPU clusters means we can generate 100x what we did at Magic Leap. Real data is abundant here too - Meta has enormous labeled and unlabeled real datasets, so synthetic data supplements rather than replaces. The infrastructure already exists: rendering pipelines, asset libraries, job scheduling, none of which we have to build from scratch. And the team size is different. We can staff multiple pods, each focused on a different aspect of synthetic data.
Why does VPS need synthetic data?
VPS needs training data for feature detection and description, image retrieval networks, depth estimation (if we go monocular), and semantic understanding. Real data comes with baked-in biases: geographic (mostly Western cities), temporal (more summer, more daytime), and condition (good weather, good lighting), plus privacy constraints around faces and license plates. Synthetic data fills exactly those gaps. Procedural city generation gets you any geography, simulation control gets you any time or weather, randomization gets you any condition, and there are no privacy concerns because there are no real people.
Architecture for Scale
┌─────────────────────────────────────────────────────────────┐
│ Synthetic Data Platform │
├─────────────────────────────────────────────────────────────┤
│ │
│ ┌───────────────┐ ┌───────────────┐ ┌───────────────┐ │
│ │ Scene │ │ Rendering │ │ Data │ │
│ │ Generation │ │ Service │ │ Catalog │ │
│ └───────────────┘ └───────────────┘ └───────────────┘ │
│ │ │ │ │
│ ┌───────────────┐ ┌───────────────┐ ┌───────────────┐ │
│ │ Asset │ │ Domain │ │ Quality │ │
│ │ Library │ │ Adaptation │ │ Validation │ │
│ └───────────────┘ └───────────────┘ └───────────────┘ │
│ │
└─────────────────────────────────────────────────────────────┘
The point of the platform shape: reusable components that any team can build on.
Lessons That Transfer
From Magic Leap: domain randomization remains essential for sim-to-real transfer, and sensor modeling has to be accurate - noise, distortion, artifacts, all of it. Validation pipelines catch bad synthetic data before it reaches training. And curriculum sampling is more effective than uniform random, as the scaling plan said it would be. The bet I made in 2018, that extreme variation would do more for sim-to-real than fidelity, holds two years and one company later.
New Opportunities
Some advantages only exist at Meta. Real assets at scale: 3D reconstructions from users can become synthetic assets, with consent. Cross-team reuse, since synthetic data built for VPS also helps Quest hand tracking, the Ray-Ban glasses work, and more. And research collaboration, meaning access to FAIR researchers already working on simulation.
Investment Roadmap
Phase 1 (Q4 2020) is a prototype pipeline for VPS-specific synthetic data. Phase 2 (2021) scales it to support VPS training needs. Phase 3 (2022) generalizes the platform for Reality Labs-wide use.
Right now the job is getting buy-in from leadership. The Magic Leap experience helps make the case.