cd ~/
LaunchDebuggingLeadershipPressureMagic Leap

Pre-Launch Crunch: Debugging Under Pressure

The final push before device launch - triaging bugs, making hard cuts, and keeping the team functional under extreme pressure.

Evyatar Bluzer
3 min read

Launch is next month. We have more bugs than we can fix, which means the job now is deciding which bugs ship with the device.

The Bug Triage Framework

Every bug gets classified. Ship blockers are the ones we cannot launch with: safety issues, data loss, crashes in more than 1% of sessions, core functionality broken. Launch quality bugs should be fixed before launch - poor user experience, performance below target, features partially working. Known issues get documented and fixed post-launch: edge cases, cosmetic problems, features that degrade gracefully.

The discipline: only ship blockers block the ship.

Where Do We Actually Stand?

Perception has 3 ship blockers in triage (tracking loss in specific lighting, a depth sensor timeout, an eye tracking crash), 12 launch quality bugs covering various accuracy and performance issues, and 47 known issues accumulated over months of edge cases. We have 4 weeks. Assuming 1 engineer-week per ship blocker on average, and knowing some blockers will spawn new blockers during the fix, we're cutting it close. "Add 50% to integration estimates" was my lesson from last year. This is the month it gets tested.

Bug triage four weeks before launchThree horizontal bars with a tier name above each and a count at its end: 3 ship blockers, 12 launch quality bugs, 47 known issues; the short ship-blocker bar is highlighted.Ship blockers: tracking loss in some lighting, depth timeout, eye tracking crashLaunch quality: accuracy and performance below target, partial featuresKnown issues: edge cases and cosmetic problems, fixed after launch3, at about 1 engineer-week each1247
Perception's open bugs four weeks out, by triage tier: 62 in total, and only the 3 ship blockers can hold the launch.

Debugging Strategy

War Room

A physical (and virtual) space where all sensor data from failing devices is accessible, experts from each subsystem sit together, and decision-making authority is in the room. No waiting for meetings or email chains.

Hypothesis-Driven Debugging

For each blocker:

  1. Reproduce reliably
  2. Form a hypothesis about the root cause
  3. Design an experiment to test it
  4. Execute and analyze
  5. If the hypothesis is confirmed, fix. If not, back to step 2.

Shotgun debugging - change random things and hope - is banned.

Failure Taxonomy

Every failure gets categorized by where it manifested (SLAM, depth, eye tracking), what triggered it (lighting, motion, user action), and why it happened (sensor issue, algorithm bug, integration error). Patterns emerge. Three "different" bugs might share a root cause.

Hard Cuts

We're cutting gesture recognition from the hand tracking pipeline (basic tracking stays, gestures don't), outdoor SLAM (indoor-only for launch), and eye tracking analytics (tracking works, metrics don't). Painful, but shipping something great beats shipping everything mediocre.

Team Health

Extended crunch is unsustainable, so we're managing it deliberately: mandatory time off after each ship-blocker fix, visible appreciation for heroic efforts, permission to escalate blockers that shouldn't be blockers, and protecting people from unreasonable asks. The launch will happen; the team needs to be intact afterward.

One month to go.

Comments