Building a Perception Team from Scratch

Specs are frozen, so the job now is execution, and execution comes down to people. I've spent most of this month building out the perception team.

The Skill Matrix

Perception sits at an unusual intersection:

The hard part is finding people who are deep in one of these areas and can still communicate across the boundaries, the translation problem I noticed in my first weeks.

Who am I actually hiring for?

T-shaped engineers, first of all: deep in one domain with a working understanding of the adjacent ones. A CV engineer who understands sensor noise models. An embedded engineer who groks the algorithms they're optimizing. I also want people who are prototype-oriented - academic publications are nice, but I want to see working code, which is why we run take-home projects instead of whiteboard puzzles.

Intellectual honesty matters more here than in most jobs. The work is hard, and I need people who can say "I don't know" and "I was wrong", because ego-driven engineers create technical debt. The last filter is system thinking: can they trace a requirement from user need through algorithm to hardware constraint? Most can't.

Team Structure

We're organizing around deliverables rather than disciplines:

Perception Team
├── Tracking (VIO/SLAM)
│   ├── CV Engineer
│   ├── Embedded Engineer
│   └── Systems Engineer
├── Depth
│   ├── Sensor Lead (optics + electronics)
│   ├── Algorithm Engineer
│   └── Test Engineer
├── Eye Tracking
│   ├── CV Engineer
│   ├── Optics Engineer
│   └── Embedded Engineer
└── Infrastructure
    ├── Simulation Engineer
    ├── Tooling Engineer
    └── DevOps

Each pod owns its feature end-to-end, and cross-cutting concerns like calibration, testing, profiling, and the simulation environment I argued for in June live with Infrastructure.

Cultural Foundations

We're establishing norms early. Weekly demos of working systems rather than slide decks. Blameless post-mortems, because things will break and we need to learn from that without finger-pointing. Direct feedback, said in the room instead of after the meeting. And documentation as an artifact: if it's not written down, it doesn't exist.

Challenges

Hiring in Israel for MR is competitive. Everyone wants the same CV/ML engineers, so our pitch is the work itself - problems that ship to millions rather than research prototypes. We're also competing with ourselves, since the US team needs the same skills, and establishing clear ownership boundaries between the sites is ongoing.

Onboarding is next month's problem.