Rendering for Perception: Why Game Engines Fall Short

Game engines optimize for "looks good at 60fps." Perception training needs "measures correctly at any cost." Those two goals sound compatible until you try to serve both with one renderer.

Where Game Engines Fall Short

Unity and Unreal are excellent at what they were built for. Point them at perception training and the shortcuts start to show. Rasterization approximates physics everywhere: shadows are fake, reflections are screen-space tricks, subsurface scattering is empirical. The built-in cameras assume a perfect pinhole or a simple lens model, with no sensor noise, no rolling shutter, no radiometric accuracy. Output is RGB only - no depth channel that matches real ToF or stereo, and no IR simulation. And every default in the pipeline, from temporal anti-aliasing to post-processing effects, is tuned to please a human eye rather than to train a neural network.

What Does a Perception Renderer Need?

Radiometric Accuracy

Light transport has to be physically correct. Energy must be conserved, BRDF evaluation must be right, and light sources need accurate modeling - area sources with falloff, not point lights.

Sensor Modeling

The render output should match sensor output. That means adding realistic noise (shot, read, fixed pattern), simulating lens distortion, vignetting, and chromatic aberration, and modeling the shutter type (global or rolling) along with exposure effects. For depth, it means simulating ToF physics or stereo matching, including the multi-path returns no calibration fully removes.

Geometric Precision

Training semantic segmentation needs perfect masks, and approximate mesh rendering won't produce them. Edges need sub-pixel accuracy, occlusion ordering has to be correct, and thin structures like hair and fences have to survive the render.

Our Rendering Stack

USD Scene Description
    ↓
Physics-Based Path Tracer (custom)
    ↓
Sensor Simulation (noise, lens, etc.)
    ↓
Training-Ready Output (RGB + depth + labels + metadata)

This is the first big technical bet of the synthetic data team we stood up in September: we're building on top of a path tracer rather than a rasterizer. Slower, but correct.

Speed vs Accuracy Trade-off

Path tracing is slow. Convergence needs 100+ samples per pixel, which for a 640x480 image means 30+ million rays minimum. We're attacking that from several directions at once: machine learning denoising lets us use fewer samples at similar quality, adaptive sampling focuses rays on complex regions, and OptiX/RTX GPU path tracing is getting fast. Where the lighting is simple we can fall back to rasterization and save path tracing for the hard cases.

Current performance: ~5 seconds per image on a high-end GPU. We need under 0.5s for practical dataset generation.

Asset Quality

Garbage in, garbage out. The rendering engine doesn't matter if the assets aren't realistic. Material parameters must be measured, not guessed, geometry needs appropriate detail, and environments need realistic clutter and weathering.

Building an asset library is a parallel effort. More on that next month.