Hand Tracking V2: Gestures, Range, and Reliability

V1 shipped basic hand tracking, and the field feedback made clear how much users noticed what was missing. For V2 the bar is hands as a primary input modality, with gesture support and reliability that rivals controllers.

V2 Hand Tracking Requirements

Two of the V1 numbers are predictions I made in public. In February 2018 I wrote that hand tracking would ship inside 15ms end to end; the network hit 12ms on target, and the system shipped at 45. The rest is everything around the network, and it is the whole story of the V2 latency line. The accuracy line has the same shape: domain randomization got the real-data error to 12mm in the lab, and the product shipped at 15mm.

Architecture Changes

Depth Sensor Upgrade

V1 ran 320x240 depth; V2 moves to 640x480. That's 4x more depth points, which buys much better hand surface reconstruction.

Additional Camera

We're considering a dedicated hand-tracking camera: positioned for an optimal hand viewing angle, running at a higher frame rate (90Hz against the 30Hz depth), focused on near-field. The trade-off is cost, power, and calibration complexity.

Neural Network Upgrade

V1 used a custom CNN with 800K parameters. V2 gets a larger model, around 5M parameters, running on the dedicated NPU - the NPU is what makes a more capable model fit within the power budget.

Gesture Recognition

Gestures are harder than tracking. A gesture is a sequence, not a frame, so we need temporal modeling - an LSTM or Transformer for context - and gesture boundaries are ambiguous, since where a gesture starts and ends is fuzzy even to humans. False positives cost more than misses: an accidental gesture is worse than a missed one, which pushes us toward high precision thresholds and confirmation mechanisms like hold or repeat. And gestures vary culturally, meaning different things around the world, so we need configurable gesture sets plus user-trainable custom gestures.

V2 Gesture Set (Launch)

1. Pinch: Thumb + index fingertip touch → select/confirm
2. Grab: Close fist → grab object
3. Point: Index extended → cursor/ray
4. Palm: Open palm facing device → stop/cancel
5. Swipe: Quick hand movement → scroll/navigate

Training Data Strategy

Real data comes from a capture rig with multiple cameras and depth sensors: 100+ subjects for diversity, controlled and natural motions, ground truth from marker-based mocap.

Synthetic data runs at 10x the scale of the real data. A procedural hand model with texture and shape variation, physics-based grasping poses, and domain randomization for robustness. A year ago I called the synthetic hand renderer the critical path because real capture could never provide millions of annotated poses. Synthetic is still the bulk of the diet, and we're building the capture rig anyway.

The third leg is semi-supervised: large unlabeled video datasets, learning from consistency (same hand, different views), and self-training on high-confidence predictions.

Interaction Design Feedback

The UX team is keeping us honest about what users actually want. Direct manipulation feels most natural, and gestures work better for mode switches than for continuous control. Hand fatigue ("gorilla arm") limits extended use. Audio or haptic feedback turns out to be essential for gesture confirmation.

The best hand tracking is invisible. Users should think about the action, never about the hand.