Building a Perception Testing Framework
Systematic testing infrastructure for perception systems - from unit tests to integration tests to field validation.
Perception systems are notoriously hard to test: the inputs are high-dimensional (images, depth maps), the outputs are continuous (poses, keypoints), and "correct" often means "good enough." One of my regrets from December was that V1 requirements existed without being systematically tested against. We built a testing framework for ours anyway, and this is how it's put together.
Testing Pyramid for Perception
/\
/ \ Field Validation
/ \ (real users, real environments)
/──────\
/ \ System Tests
/ \ (full pipeline, recorded data)
/────────────\
/ \ Integration Tests
/ \ (component boundaries)
/──────────────────\
/ \ Unit Tests
/ \ (individual functions)
/────────────────────────\
Each level catches different bugs, and no level substitutes for another.
Unit Tests
At the bottom sit individual functions: a feature detector given an image patch should find the correct corners, a depth filter given noisy depth should produce filtered depth, a pose optimizer given constraints should converge to the optimal pose. Simple in principle, with perception-specific complications in practice - floating point comparisons need tolerances, anything with random initialization needs seeded tests, and performance has to be tested alongside correctness.
We have 2,000+ unit tests. They run on every commit.
Integration Tests
One level up, we test component boundaries: the camera driver delivering images to the feature extractor, the feature extractor feeding SLAM, SLAM updating the pose service. Dependencies get mocked so the tests can focus on data flow and error handling. The handoffs we watch most closely are sensor to algorithm, algorithm to algorithm, and algorithm to API surface, because that's where assumptions quietly diverge.
System Tests
Full pipeline on recorded data. The regression datasets collect our most challenging sequences - low light, fast motion, dynamic scenes, multi-room walks - plus the edge case library we assembled from V1 field failures, with ground truth from motion capture or surveyed markers. The metrics are Absolute Trajectory Error (ATE), Relative Pose Error (RPE), tracking loss events, and feature count over time. An automated dashboard tracks everything across commits, and regressions block merge.
Field Validation
At the top: real devices in real environments, meaning beta users with instrumented builds, telemetry aggregation, and failure classification, the loop that started with V1's post-launch telemetry. Field validation catches what lab testing misses - the edge cases we didn't imagine, the environmental factors we don't control, the user behaviors we didn't anticipate. Every framework needs this layer, because your imagination is not a test plan.
Can We Generate Tests Automatically?
Partly. Fuzz testing throws random inputs at components to find crashes: random images into the feature detector, which should never crash, and random depths into the mesh builder, which should degrade gracefully. Property-based testing defines invariants and generates cases against them. Tracking should be consistent, so running forward then backward should recover the original pose. Depth filtering should not increase noise. Pose optimization should decrease error.
Flaky Tests
Perception tests flake in ways ordinary software tests don't, because numerical precision varies across platforms, multi-threaded code has race conditions, and random initialization causes variance. The fixes are unglamorous: deterministic random seeds, tolerance-based comparisons, retry with logging on failure, and quarantine for persistent offenders.
Current flake rate: 0.3%, down from 5% a year ago.
CI/CD Integration
A commit triggers unit tests (2 min). A PR runs unit plus integration (15 min). A merge kicks off full system tests (2 hours), and a daily job runs the extended regression suite (8 hours).
No human should need to run tests manually for routine development.