Hand Tracking for AR: From Depth to Skeleton

Hands are the natural interface for AR - no controllers to find, no buttons to learn, you just reach out and interact. Making that work, the last item on my 2018 list, means solving one of the hardest perception problems there is.

Why Are Hands So Hard to Track?

Self-occlusion, first of all: fingers constantly block each other from the camera's view. Then the configuration space. A hand has 25+ joints, each with multiple degrees of freedom, so the space of possible poses is enormous. Hands also move fast, up to 5m/s in gestures, which forces low latency tracking. Appearance varies with skin tone, hand size, jewelry, and nail polish. And most interactions happen at arm's length, 30-60cm from the headset.

The Pipeline

Depth Image → Hand Segmentation → Keypoint Detection →
Skeleton Fitting → Temporal Filtering → Output Pose

Hand Segmentation

First, find the hands in the scene. We combine depth thresholding (hands are typically at a known range), a learned segmentation network for precise boundaries, and temporal tracking to maintain identity across frames.

Keypoint Detection

From the segmented hand region we detect anatomical keypoints: 5 fingertips, 10 finger joints, 5-10 palm points, and 2 wrist points. There are two ways to do this - heatmap regression, where a CNN outputs probability maps for each keypoint, and direct regression, where the CNN outputs (x,y,z) coordinates. Heatmap is more robust; direct is faster. We use heatmap.

Skeleton Fitting

Keypoint detections are noisy and may be partially occluded, so we fit a kinematic skeleton model on top: known bone lengths (calibrated or estimated from visible segments), joint angle constraints (fingers don't bend backward), and temporal smoothness priors. The optimization minimizes keypoint reprojection error subject to those kinematic constraints.

Temporal Filtering

Raw per-frame outputs are jittery. We apply Kalman filtering for position and velocity estimation, occlusion-aware interpolation when confidence drops, and gesture-specific smoothing. A pinch should be crisp; a wave should flow.

Depth vs RGB

We could do hand tracking with RGB alone; many phone implementations do. But depth gives you unambiguous 3D with no scale uncertainty, works in low light, and cares less about skin tone. The trade-off is depth sensor power, depth artifacts at hand edges, and the 320x240 resolution we froze in 2016 knowing it might limit hand tracking. For our system, depth-first makes sense, with RGB as backup for outdoor scenarios.

Training Data

This is where synthetic data shines. We need millions of hand poses, covering occlusions, lighting variation, and backgrounds, all with perfect joint annotations. Real data collection can't provide that, the same arithmetic that justified the synthetic data team, which makes our synthetic hand renderer a critical path item.

Next month: the neural network architecture itself.