SLAM in Low Light: Pushing the Sensor Limits
Techniques for maintaining visual SLAM in challenging lighting conditions - from sensor improvements to algorithmic adaptations.
Low light kills visual SLAM. Fewer photons means noisier images, noisier images mean fewer features, and fewer features mean tracking failures. V2 has to handle rooms lit by candles, so this is now my problem.
The Low-Light Challenge
Some numbers to frame it. At 1 lux, roughly a moonlit night, a standard camera collects about 10 photons per pixel per frame, SNR sits under 10dB, and the signal is buried in noise; feature detection returns mostly false positives. At 100 lux, dim indoor lighting, you get around 1000 photons per pixel per frame and roughly 30dB of SNR - usable but not great, with feature detection functional but degraded.
The current system fails below about 30 lux, which is the dark half of the lighting problem the field data flagged. The V2 target is reliable tracking at 3 lux.
Sensor-Level Improvements
Larger Pixels
More photon collection area, at the cost of pixel count. V1 uses 3μm pixels at 640x480; the V2 option is 4μm pixels at 1280x720, the tracking camera line in the V2 sensor table. Net: 4x more light per pixel and 4x more pixels total, so 16x more photons.
Higher Quantum Efficiency
Better conversion of photons to electrons. Standard silicon sits at 50-60% QE; backside illuminated (BSI) sensors reach 70-80%. That's +30% more signal.
Lower Read Noise
Modern sensors are achieving under 1e- read noise against V1's 2-3e-. This matters because at very low light, read noise dominates, and lower noise means usable signal at lower photon counts.
Global Shutter Considerations
Global shutter sensors typically carry more noise than rolling shutter, but rolling shutter produces artifacts under fast motion. We need global shutter for tracking, so we accept some noise penalty.
Algorithmic Improvements
Sensors only get us partway; the algorithms have to adapt too. Feature detection comes first: lower the detection thresholds and let matching filter out the bad features, use larger patches so descriptors carry more pixels and survive noise, and detect at multiple scales.
Tracking strategy changes with it. Trust the IMU more when vision is poor, track feature lifetimes so features get reused across more frames, and fall back to a degraded mode that reduces update rate and increases integration time.
There's also the learning-based route. Trained feature detectors - SuperPoint and D2-Net style architectures, trained on real plus synthetic low-light data - may outperform hand-crafted ones in low light. We're exploring this for V2.
IR Illumination Option
Could we add IR illumination for tracking? It would solve low light completely, but it costs power and battery, and it interferes outdoors, the same sunlight arithmetic the depth illuminator fights. Current direction: passive cameras, with active assist for extreme cases.
Testing Protocol
The new low-light test suite runs in a 1 lux controlled environment, uses calibrated neutral density filters for repeatability, and follows standardized motion profiles. Pass criteria: under 3 tracking losses per 10 minutes at 3 lux.
V2 sensors arrive for evaluation next month.