cd ~/
Domain RandomizationSynthetic DataSim-to-RealDeep LearningMagic Leap

Domain Randomization: Brute-Forcing the Reality Gap

Using extreme variation in synthetic data to bridge the sim-to-real gap - theory, practice, and hard-won lessons.

Evyatar Bluzer
3 min read

Models trained on beautiful synthetic data keep failing on ugly real sensor images. The reality gap has been haunting us for months, and domain randomization is our main weapon against it.

The Core Idea

If you can't make synthetic data perfectly match reality - and even a physically based path tracer doesn't - make it match everything:

"If the model has seen enough variation in simulation, reality is just another variation."

So we randomize textures (including unrealistic ones), lighting (extreme conditions), noise (more than real sensors produce), geometry (within plausible bounds), and camera parameters (beyond spec). The model learns features robust to all of these variations, including the specific variation called "real data."

What We Randomize

Visual Properties

Textures get random colors, procedural patterns, and photo textures. Lighting varies in direction, intensity, color, and number of sources. Shadows range from hard to soft with varying direction and intensity, and backgrounds run from uniform colors to complex scenes.

Sensor Properties

Noise comes in Gaussian, Poisson, and salt-and-pepper varieties, pushed beyond realistic levels. On top of that we add motion and defocus blur in varying amounts, under and over-exposure, and JPEG-like compression artifacts.

Geometric Properties

Object scale varies +/-20% from nominal, positions get jitter and displacement, and viewpoints cover a broader range than we expect in deployment.

Hand-Specific

Skin tone spans the full spectrum, including unrealistic colors. Hand shape varies in scale, finger lengths, and joint angles, and we throw in accessories: rings, watches, sleeves.

How Much Randomization Is Too Much?

More is not always better. Too little and the model overfits to the synthetic domain. Too much and it can't learn meaningful features at all; it sees only noise. The sweet spot has enough variation to be robust without losing the signal, and we tune the magnitude per-factor using validation on real data.

Curriculum Strategy

Some factors are better introduced gradually. We start with realistic rendering, add noise factors, then geometric variation, and only then extreme texture randomization. The curriculum lets the model learn basic features before confronting the extreme stuff.

Results

The numbers for the heatmap keypoint detector:

  • Trained on synthetic only (no randomization): 45mm error on real data
  • Trained on synthetic with randomization: 12mm error on real data
  • Trained on real data: 8mm error on real data
Keypoint error on real data by training setThree horizontal bars with a label above each and a millimetre value at its end: 45 mm for synthetic only, 12 mm for synthetic with randomization, 8 mm for real data; the middle bar is highlighted.Trained on synthetic only, no randomizationTrained on synthetic with domain randomizationTrained on real data45 mm12 mm8 mm
Keypoint error on real data by training set: randomization takes synthetic-only from 45 mm to 12 mm, 4 mm short of training on real data.

Domain randomization closed 85% of the gap. The remaining 15% comes from distribution mismatch in poses, subtle artifacts our randomization doesn't capture, and the implicit regularization effect real data seems to have.

Failure Modes

Domain randomization can't fix systematic biases in the synthetic data (always-centered objects, say), factors of variation we never modeled (motion blur patterns we missed), or out-of-distribution inputs unlike anything in the randomization range. So this never really ends: as we discover failure cases, we add new randomization factors.

Comments