Models Almost Never Ask
A new benchmark shows LLM agents almost never ask before committing memories - and why the write policy is the ambient wearable's real moat.
Somewhere in the pipeline of every ambient wearable there is a decision point that determines whether the thing you just said becomes a permanent fact about you. A preprint that landed on arXiv on August 20 is the first benchmark I have seen that tests whether models make that decision well. They mostly do not.
The short version: the MCB benchmark (arXiv:2608.19564, August 2026) tests whether an LLM agent should persist information, keep it in the current context, re-verify it, or ask the user - and finds that models almost never ask. Few-shot prompting lifted overall accuracy from 0.557 to 0.771 while clarification recall stayed stuck at 0.333. For an always-on wearable, which cannot afford to interrupt its user, the fix is a write policy that corroborates across days before committing anything to the life-context layer.
The Write Decision Finally Has a Benchmark
"Remember, Verify, or Ask? Cross-Family Evaluation of Memory Commitment in LLM Agents" (Baichuan Li, Junyi Yao, Zihao Zheng, arXiv:2608.19564) is a small paper about a large hole. Everyone benchmarks memory retrieval - can the agent find the fact it stored. Almost nobody benchmarks memory commitment: the upstream decision about whether a piece of interaction-derived information deserves to be stored at all, held only for the current session, re-verified because it may have changed, or bounced back to the user as a clarifying question.
The authors build a 140-scenario benchmark around this four-way decision, split 70 development and 70 held-out, plus a 70-item contrast set, and they evaluate both the model's stated choice and its actual tool-call selection across Claude and Qwen model families. The labeling is careful for a preprint: two non-author annotators, 97.1% agreement, Cohen's kappa of 0.962. The headline finding is an asymmetry. Models are reasonably willing to re-verify facts that might have changed, and nearly incapable of asking a human to resolve ambiguity:
bare Qwen asks on 0/12 clarification items while verifying 12/18 freshness items
And prompting does not buy your way out:
Few-shot prompting raises accuracy from 0.557 to 0.771 (paired delta = +0.214, Holm-adjusted exact McNemar p_H = 0.002), yet clarification recall remains 0.333.
Clarification recall is the fraction of cases where the right move was "ask the user" and the model actually asked. Two thirds of the time, the right move was a question and the model did something with the information on its own instead.
What the Numbers Do and Do Not Show
I should scope this properly: arXiv is unreachable from the network I am writing on, so I am working from the abstract and the reported results rather than the full tables. Read them yourself before betting a roadmap on any single number.
Even taken at face value, that 0/12 is twelve items. Seventy held-out scenarios split across four action categories leaves single-digit denominators everywhere, so the per-category rates have wide error bars, and I could not find a code or dataset release to check against. Two model families is "cross-family" in the technical sense and nothing like coverage of the frontier.
The bigger gap is between the benchmark's world and mine. MCB scenarios are typed chat, where the user explicitly tells the agent things and a clarifying question costs one message. An always-on wearable lives in the inverse world. It overhears rather than being told: sarcasm, a third party's opinion, a podcast playing in the background, your half of an argument you did not mean. The ambiguity rate is far higher than anything in a chat benchmark, and the budget for asking is close to zero. In other words, the paper measures the easy version of the problem and models still fail it.
Why Can't the Ambient Assistant Just Ask?
Because on a wearable, every clarifying question is an interruption, and interruption is the scarcest budget an ambient product has. Calm tech has said this for two decades: a device that pings you to confirm what it overheard is a device you take off.
If clarification is off the table, the write policy has to replace it with something else, and the honest answer is corroboration over time. A fact overheard once goes into a provisional buffer with provenance attached - who said it, when, at what confidence. It only graduates into the durable life-context layer when independent observations across days agree with it, and it stays correctable, because the user will eventually see the coach act on it. The equivalent lesson at Magic Leap came from spatial persistence: users forgave the system for forgetting where a virtual object was, and did not forgive it for confidently remembering wrong. Amnesia is a bug. Confabulation is a breakup.
A relationship coach that cites a "fact" about your marriage which was actually a joke, or your sister's opinion, has not made a small retrieval error. It has broken the alliance the entire product depends on.
The Write Policy Is the Moat
Every player in this market competes on capture and retrieval: more hours, better transcripts, faster search over your life. Nobody is shipping a write policy, and until August nobody could even measure one. That is the why-now: the commitment decision is finally legible as a benchmark, and the first published numbers say the default behavior of frontier models is to decide on their own rather than ask.
Why is it hard to replicate? Because a good write policy is trained on correction signals, and correction signals only come from users wearing the device for months and telling the coach where it was wrong. That data does not exist in any public corpus and cannot be cold-started - the same moat I have written about for the context layer, one level deeper in the stack.
Every wearable pitch deck I see has a slide about how much the device remembers. I am still waiting for the one with a slide about what it refuses to.