cd ~/
Eye TrackingOpticsHardware DesignIlluminationMagic Leap

Eye Tracking Optics: Imaging the Eye in a Headset

The optical design challenges of capturing the eye from millimeters away - illumination, imaging, and integration with the display system.

Evyatar Bluzer
3 min read

Imaging the eye from inside a headset has very little in common with webcam-style eye tracking. Everything is closer, faster, and more constrained.

Geometric Constraints

The camera sits 15-25mm from the eye and has to cover a 12mm x 8mm eye box to account for headset slippage, which pushes the camera FOV to roughly 60° just to image the entire eye across that box. At this proximity, depth of field is extremely shallow, holding focus across the full eye from cornea to sclera is a real problem, and lens distortion is severe.

Illumination Architecture

Controlled illumination is what makes the rest of the pipeline possible. IR images enable pupil and iris boundary detection, glints provide the geometric reference, and the LEDs must overpower ambient IR, since sunlight carries strong NIR content.

The design space has a few axes. LED position first: on-axis, with the camera co-located with the LEDs, gives you the bright pupil effect from retro-reflection but requires a beam splitter, while off-axis LEDs around the display give dark pupil imaging and simpler optics at the cost of asymmetric illumination. LED count next: more LEDs mean more glints and more geometric constraints for gaze. Two LEDs is the minimal system, pupil plus one glint pair; four or more is overdetermined and more robust; and every LED added costs power and risks stray light. Finally modulation, synchronizing the illumination with camera exposure, which enables ambient rejection and lets you identify which LED produced which glint.

We're using 4 off-axis LEDs with temporal modulation.

Eye camera and illumination layoutA side section: an eye circle at left with a dashed eye box in front of the pupil, a vertical display plate to the right with an LED dot at each end and dashed rays to the eye, and a small camera box below the display whose highlighted 60 degree cone covers the eye. ~60° field of view eye box 12 x 8 mm where the pupil sits as the headset slips display waveguide off-axis IR LEDs, 4 around the display rim, temporally modulated two shown in this section eye camera, 15-25 mm from the eye
The eye imaging layout in section, drawn to scale from the numbers above: a camera 15 to 25 mm from the eye needs roughly a 60 degree cone to keep the whole eye in view across the 12 by 8 mm eye box, with the IR LEDs off-axis around the display rim.

Camera Selection

The specs that matter:

  • Resolution: 320x320 is sufficient if we're pupil-limited anyway
  • Frame rate: 90-120Hz for smooth tracking through saccades
  • Shutter: global shutter, because the eye moves during exposure
  • Sensitivity: high quantum efficiency at 850nm
  • Size: it has to fit the headset form factor

As for CMOS vs CCD, CMOS wins on power and integration, and global shutter CMOS parts are available now.

At spec freeze I put eye camera resolution on my list of worries, traded away for power and likely to hurt pupil tracking in bright light. So far it hasn't earned its place there; we're pupil-limited before we're pixel-limited, and 320x320 is enough.

Can the display waveguides do double duty?

One innovation we're exploring is diffractive optical elements (DOEs) for eye imaging. The display already uses waveguides to project images, so can the same or similar structures direct the eye-tracker camera off the optical path, create structured illumination patterns, and multiplex imaging and tracking functions? If so, the overall optical architecture gets dramatically simpler. Patent work is ongoing.

Integration Challenges

Everything eye tracking adds has a price. Cameras near the user's eyes raise comfort and aesthetics questions, IR illumination triggers eye safety certification under the same Class 1 limit the depth illuminator lives with, the processing load hits the power budget, and there's one more calibration to run. Justifying all of it requires clear value delivery: the power that foveated rendering saves must exceed the power that eye tracking costs. The math works, but the margins are thin enough that execution decides it.

Comments