cd ~/
Generative AILLMSpatial ComputingVPSMeta

Generative AI Meets Spatial Computing

How large language models and generative AI are changing spatial computing - from scene understanding to content creation.

Evyatar Bluzer
3 min read

LLMs have scrambled a lot of assumptions in my corner of the field. I work on visual positioning, and the question I keep turning over is how much of the classical spatial pipeline survives contact with models that can describe a scene in plain language.

How Spatial AI Worked Before LLMs

Pre-LLM spatial understanding was a stack of specialized models, one per pipeline stage. Feature detection meant SIFT or a learned descriptor. Geometric reasoning meant SLAM and SfM. Object recognition meant a CNN classifier trained for exactly that task. Each one needed its own model and its own training data, and none of them shared anything with the others.

LLM-Era Opportunities

Semantic Understanding

LLMs can interpret what's in a scene:

Image → Vision-Language Model → "A busy café with outdoor seating"

Rich semantic understanding without task-specific training. That used to take a bespoke model per task.

Scene Description for Localization

Instead of feature matching:

Query: "I see a red brick building with a blue awning next to a parking meter"
Match: Find locations matching this description

Language becomes the query interface to the spatial database.

Contextual Awareness

An LLM-powered AR assistant:

User: "Where should I sit?"
System: [Uses VPS for location] + [Uses LLM for reasoning]
Response: "The table by the window has the best view and is in shade"

Neither half is useful alone here. VPS knows where you are but cannot reason about it, and the LLM can reason but has no idea where you are standing. Put together, they answer a question a person would actually ask. This is the contextual-AI use case I sketched for glasses with the reasoning half filled in.

Technical Integration

VPS + Vision-Language Models

Camera → VPS (where am I?) → Scene understanding (what's here?) →
LLM reasoning (what does it mean?) → User value

Each component does what it is best at.

Challenges

Latency is the first wall: LLMs are slow, and spatial computing needs real-time answers. Cost is the second, since LLM inference is expensive and you cannot run it on every frame. Grounding cuts the other way - LLMs hallucinate, and spatial ground truth gives you a check against that.

VPS and LLM pipelineSequence diagram with five lifelines, Camera, VPS, Vision-language model, LLM reasoning and User, connected left to right by four solid arrows and one dashed arrow, with a note on each middle lifeline; the note on the LLM lifeline is highlighted. CameraVPSVision-language modelLLM reasoningUser frames location scene description spatial ground truth,a check against hallucination "The table by the window hasthe best view and is in shade"Where am I?coordinates from thefeature databaseWhat's here?"A busy café withoutdoor seating"What does it mean?slow and expensive:not on every frame
The VPS plus LLM pipeline as a handoff: camera frames become a location, the location becomes a scene description, and the LLM turns the description into an answer, with VPS also supplying the ground truth that checks the LLM.

Experiments Underway

We're prototyping LLM-described location matching (language-based VPS), generative scene completion to fill in unmapped areas, and conversational spatial search along the lines of "find me a coffee shop with seating". Early results are promising. None of it is production-ready.

Where Does VPS Go From Here?

VPS might evolve from a feature database into a semantic scene database, from coordinate output into contextual understanding, from a developer API into an end-user experience. The goal stays put - help devices understand where they are. The methods may change dramatically.

Personal Interest

This intersection of spatial AI and generative AI is where I want to be. I've spent years on the VPS side, and the LLM side is moving too fast to ignore; the overlap between the two feels underpopulated. I'm starting to think about what that means for the next chapter I was circling in December.

Comments