Keep six minutes of video in a model's context the obvious way and it costs about 62,000 tokens, 25 GB of GPU memory, and four and a half seconds before the first word of an answer. Keep instead the captions the model wrote while the video was playing, and the same six minutes costs 4,308 tokens and about a tenth of a second. That gap is the whole case for how an always-on wearable will remember a day, and OneStreamer (arXiv:2610.01762 (opens in a new tab)), out of Nanjing this month, is the cleanest demonstration of it I have read.
TL;DR
- OneStreamer (arXiv:2610.01762) is a 4-billion-parameter streaming video model that writes timestamped captions as video plays and answers later from the text, never re-reading the pixels it already captioned.
- On one six-minute clip, answering from captions instead of retained frames cut context from 62,094 tokens to 4,308, GPU memory from 25 GB to 10 GB, and time to first token from 4.6 seconds to 0.12.
- The same caption stream drives both jobs an ambient assistant has: remembering what happened and deciding when to speak.
- Its when-to-speak training supervises 27.5 percent of the labeled decision points and still beats labeling all of them, so the proactivity signal is cheaper to teach than the field assumed.
- The compute is a datacenter GPU, not a frame of glasses, and the memory is only ever as good as what the model thought to caption while it was happening.
The memory is the captions, not the frames
OneStreamer keeps a small visual window - the last sixteen frames, first in, first out - and as events complete it writes them down as text: local captions of what is happening, then short summaries of finished events. Once a frame drops out of that window, the model never looks at it again. What survives is the sentence it wrote, carried forward in the text context while the pixels are gone. The authors call this Proactive Hierarchical Caption Memory, and the name earns itself: the memory is hierarchical, with details rolled up into event summaries, and it is written proactively, before any question that might need it exists.
That is the only design that fits a day of egocentric video into something a device can hold. A day of raw video is terabytes; a day of captions is a text file. The repository's measurements put the trade on one six-minute clip: holding the full visual history costs 62,094 context tokens and a 4.56-second wait before the first word; holding the captions the model already wrote costs 4,308 tokens and 0.12 seconds. Fourteen times less context, and the reply starts almost before you finish asking.
None of it is free. A caption is a lossy summary, and you get to write it once, while the event is happening, before you know what anyone will ask. The benchmark that showed agent memory breaks at write time rather than at recall was measuring exactly this on text; OneStreamer carries the same failure to video and does not escape it. If the caption said "two people talking" and tomorrow you need to know which of them looked away first, the frame that held the answer is already gone.
Why are the memory and the when-to-speak the same file?
Because the captions are also the signal the model reads to decide it has seen enough to respond. OneStreamer does not run a memory module beside a separate proactivity module; the text it writes to remember is the text it watches to time its reply. I argued for audio that a wearable should stay quiet and speak only when the room gets something wrong, and the open question there was what the device watches to make that call without running a live video model every second and draining the battery by noon. The answer here is that it watches its own notes.
The training result sharpens it. The proactive half is taught by marking when the model should shift from watching to answering, and the authors supervise only 27.5 percent of those labeled transition points, yet beat supervising all of them and beat a random baseline given the same budget. So the when-to-speak lesson is sparse: you do not label every moment, only the few that turn. The expensive part of an ambient assistant was supposed to be teaching it when to interrupt, and this says that lesson needs far less labeled data than anyone budgeted for. Memory and timing come out of one artifact, trained mostly on the cheap.
On the benchmarks the 4B model leads its base everywhere it is measured - streaming question answering, response timing, online video understanding - but lead is the word, not solved. Its proactive-interaction score (OmniMMI) is 36.6, up from 29.4. Its proactive-response score is 48.7. Best in class and still under half: the model knows when to speak better than anything else open, and it is wrong most of the time.
Where it breaks
The compute, first. Those elegant token counts are measured on an H200, a datacenter accelerator, and even the lean caption path spends 10 GB of GPU memory to answer one question about six minutes of video. Streaming and on-device are not the same word. Nothing in this paper runs on a frame of glasses or a pendant. What would run on the wearable, if the design ships, is the captioner - a vision model writing a sentence every few seconds, all day, off a coin cell - and that cost appears nowhere in the results. The paper proves the memory is cheap to query. It says nothing about whether the thing that writes it is cheap to run.
Then the catch underneath the whole category: the memory is only as good as the caption, and the caption is written blind to the future. A richer visual window or a bigger model does not fix that, because the information was discarded at write time on purpose. This is the same wall every passive recorder hits, now with a token budget attached to it. I should also be honest about my sources: I could not reach the full paper - arxiv.org and the paper page sat behind the egress block these routines run under - so every number above is from the authors' public GitHub repository (opens in a new tab), and I am scoping to what it reports rather than grading claims in text I could not open.
So here is where I think it lands, with a date on it. Within a year the memory feature on camera glasses, Meta's included, stops being "we store your video" and becomes "we store the sentence that described your video," because the sentence is fourteen times cheaper to keep and to search and nobody wants the terabytes. That quietly moves the privacy question. The fight over camera glasses fixated on whether the lens was recording; the next one is about the caption - a text log of your day, written by a model, far easier to store, index, and subpoena than any video ever was. A frame is evidence of a moment. A caption stream is a diary someone else kept on you.
Two calls, then. By the end of 2027, at least one shipping always-on visual-memory wearable stores its long-horizon memory as text written during capture and queried as text, not as retained frames re-encoded when you ask. And I will be surprised if any open streaming video model reports an OmniMMI proactive-interaction average above 50 before then. The remembering got cheap this month. The timing is still the hard part, and 36.6 is where the frontier actually sits.