Scaling Synthetic Data: From Thousands to Millions
Building infrastructure to generate millions of training images - the engineering of a synthetic data factory.
At our current rate of 1000 images per day, the hand tracking dataset V2 needs alone would take 14 years to generate. In December 2017 I put "scale synthetic data generation 10x" on the list for 2018. It did not happen. The rate hasn't moved, and the target has grown to 100x. The synthetic data concept is proven; the prototype scale is not. We need to go from thousands of images to millions per training run.
Scale Requirements
Training a robust perception model needs:
- Hand tracking: 5M+ images across poses, lighting, backgrounds
- Eye tracking: 2M+ images across gaze directions, face shapes
- Scene understanding: 10M+ images across environments, objects
The Scaling Challenge
Compute
Rendering is the bottleneck, at about 5 seconds per image on a high-end GPU. In October 2017 I wrote that render cost had to fall from 5 seconds to under half a second before dataset generation was practical, and that ML denoising, adaptive sampling, and RTX would get us there. It's still 5 seconds. Denoising is still on the plan below; the lever we're actually pulling is a thousand GPUs. The plan: burst to 1000 GPU instances in the cloud for render jobs, optimize the renderer itself (denoising lets us get away with fewer samples), and drop fidelity where it doesn't matter, since not every image needs ray tracing. Target: 100,000 images per day, sustained.
Asset Pipeline
Diverse training needs diverse assets - 1000+ 3D environments, 10,000+ objects, 500+ hand textures and shapes, and unlimited procedural variations on top of those. The pipeline that feeds this combines automated acquisition from 3D repositories, procedural variation of base assets, and quality validation gates.
Variation Management
How do you sample from a trillion possible configurations? Stratified sampling ensures coverage of the factors we already know are important. Curriculum sampling starts uniform, then focuses on failure cases. And active learning lets the model tell you what it struggles with.
Metadata and Versioning
Every image carries its exact scene configuration for reproducibility, ground truth labels that fall out of the render automatically, and the variation parameters for later analysis. Storage and tracking for millions of images turns into a database engineering problem in its own right.
Cost Management
Cloud GPU rendering is expensive. 1000 GPUs at $2/hr for 24 hours is $48,000 per day, and 100M images comes to $4.8M in compute alone. So cost reduction is part of the architecture: spot instances save 70% on fault-tolerant workloads, render optimization compounds (2x speed is 50% cost), and smart sampling buys better coverage with fewer images.
The annual synthetic data budget is $2M. We have to make it count.
Validation
How do we know the synthetic data is any good? Sampled renders get reviewed by humans, and distribution statistics get compared against real data. We're building both checks into the pipeline as automated gates, so no bad data reaches training. But the test that settles it is model performance. Does training on it work?