Transfer Learning for Perception: Sim-to-Real and Beyond

Everything we train starts life on synthetic data, the pipeline we are scaling toward 100K images a day, and then has to survive contact with real devices. That transfer step carries the weight of our entire synthetic data strategy, so we spend a lot of effort making it hold.

The Transfer Challenge

Train a model on synthetic data and it hits 95% accuracy on the synthetic test set. Point the same model at real data and it drops to 72%. That 23% difference is the sim-to-real gap, the one we have been brute-forcing with randomization since 2018, and the synthetic strategy stands or falls on how well we close it.

Domain Adaptation Techniques

Feature Alignment

The idea is to force the network into domain-invariant features. The adversarial version adds a discriminator that tries to tell synthetic features from real ones while the encoder trains to fool it:

              ┌──────────────┐
Image ───────►│   Encoder    │───► Features ───► Task Head ───► Prediction
              └──────────────┘         │
                                       │
                              ┌────────▼────────┐
                              │  Discriminator  │
                              │ (syn vs real)   │
                              └─────────────────┘

Loss = TaskLoss - λ × DomainLoss

The subtraction makes the encoder adversarial to the discriminator. The non-adversarial alternative is Maximum Mean Discrepancy (MMD), which minimizes the statistical distance between the two feature distributions directly.

Self-Training

Self-training bootstraps from the model's own predictions on unlabeled real data:

1. Train on synthetic (labeled)
2. Apply to real (unlabeled), get predictions
3. Filter high-confidence predictions
4. Retrain on synthetic + pseudo-labeled real
5. Repeat

Each iteration improves real-domain performance. No human labels the real data at any point, which is the appeal.

How Much Real Data Do You Need?

We measured it:

• 0% real: 72% accuracy
• 1% real + 99% synthetic: 85% accuracy
• 10% real + 90% synthetic: 91% accuracy
• 100% real: 93% accuracy

The first slice of real data does wildly disproportionate work, and the curve flattens fast after that. So we collect strategically instead of exhaustively.

One correction to the record. In 2017 I committed to synthetic data producing better models than pure real-data training by the end of 2018. By these numbers it has not: 10% real plus 90% synthetic lands at 91% against 93% for all-real. What it did do is make a small real slice worth most of the gap, which is a different result and a more useful one.

Cross-Task Transfer

Can training on one task help another? Low-level features like edges and textures transfer across tasks, so shared representations pay off. Our concrete case went from a hand segmentation model to a hand keypoint model: pre-train the encoder on segmentation, where labels are abundant, then fine-tune the full model on keypoints, where labels are scarce. That produced 15% better keypoint accuracy with the same keypoint data.

Multi-task learning pushes the same idea further, with a shared encoder feeding multiple heads for segmentation, depth, and keypoints. You get a regularization effect, efficient use of data, and a single model serving multiple needs. You also inherit task interference, where improving one task hurts another, and the perennial argument over loss weighting.

Practical Pipeline

Our production pipeline:

1. Pre-train large model on synthetic data (all the data we can generate)
2. Domain adapt using adversarial + self-training (no real labels needed)
3. Fine-tune on curated real dataset (expensive to collect)
4. Specialize per-device if calibration data available

Each stage improves real-world performance, and each one costs more per sample than the one before it, which is exactly why the ordering matters.

Measuring Transfer

Three numbers tell us whether any of this works: the absolute gap (real accuracy minus synthetic accuracy), the transfer ratio (real accuracy with transfer divided by real accuracy with real training), and data efficiency (real samples needed to reach target accuracy).

Our hand tracking model sits at a 0.85 transfer ratio with zero real data. With 10K real images it reaches 0.97.

Synthetic data is working.