Eye Tracking in AR: Technical Challenges and Approaches

Foveated rendering, natural UI, social presence - the next level of AR interaction runs through eye tracking, which also happens to be one of the hardest perception problems on the headset, and the last of the three I signed up for to get its own post.

Why is eye tracking so hard?

Start with the eye itself. It's a moving target, with saccades reaching 500°/s, and the tracker has to keep up. Conditions vary wildly: pupil size swings between 2 and 8mm depending on lighting, and makeup, glasses, and contact lenses all add variation. The optics are unusual too, since we're millimeters from the eye rather than across a desk like webcam trackers, which means extreme wide-angle distortion. Eyelids, eyelashes, and reflections from the display all interfere. And the data is biometrically sensitive - iris patterns are unique identifiers, so the privacy constraints from last month apply to everything downstream.

The Eye Tracking Pipeline

IR Illumination → Eye Camera → Pupil Detection →
Glint Detection → Gaze Estimation → Filtering/Prediction

IR Illumination

Multiple IR LEDs create "glints", corneal reflections that provide geometric reference points.

Pupil Detection

The task is finding the pupil ellipse in the eye image, which sounds simple until you account for variable size and shape, partial occlusion by eyelids, and reflections from the display. The classical approach is edge detection plus ellipse fitting; the learning approach is a trained pupil segmentation network.

Glint Detection

Glint positions relative to the pupil indicate gaze direction. The catch is that display reflections create false glints, which we discriminate using modulation patterns.

Gaze Estimation

A model-based estimator fits a 3D eye model - cornea as a sphere, pupil as a disk - and takes the optical axis as gaze. It requires calibration per user, is robust once calibrated, and handles glasses poorly. An appearance-based estimator regresses directly from the eye image to a gaze vector; it needs large training data, can absorb more variation, and may not generalize to unseen conditions. We're going hybrid, model-based geometry with learned refinement.

The Calibration Problem

Every user is different: eye shape, kappa angle (the offset between visual axis and optical axis), head-eye geometry. The standard solution is a 5-9 point calibration where the user looks at known targets. The UX problem is that users hate it. Calibration is boring, takes time, and has to be repeated, so we're researching implicit calibration, inferring the parameters from natural gaze behavior over time.

Foveated Rendering Requirements

To save rendering compute, you only render full detail where the user is looking. That takes latency under 10ms from eye movement to render adjustment, accuracy under 1° to avoid visible quality transitions, and prediction of saccade endpoints, because saccades are faster than rendering. That one-degree accuracy figure is the answer to the accuracy question I left open in November, at least for foveation. This is aggressive. Our current system achieves ~15ms latency, and getting below 10ms requires tight integration with the display pipeline.

The optics behind all of this get their own post next month.