Someone takes a breath, pauses half a second, and says "fine." The transcript records one word: fine. The inhale before it, the flatness of the delivery, the laugh two sentences earlier that came a beat too fast - none of that survives the trip from microphone to text. For a meeting note-taker that loss is a rounding error. For a wearable that claims to understand how you and the people around you actually get along, it is most of the signal, gone at the first step.
A speech-event detector posted this month (arXiv:2610.14232) is the first open model I have seen that tries to put the discarded layer back, and then measures, without flinching, how far that gets. The release (opens in a new tab) is 5.8 million parameters, small enough to run on a pendant, and it tags 54 event classes every 40 milliseconds: 20 things a speaker's body does - breath, sigh, laugh, cough, a sharp gasp - and 34 sounds in the room around them. The interesting number is not the parameter count. It is the accuracy, and the accuracy is where the paper earns its keep.
The short version: A speech-event detector (arXiv:2610.14232) is a 5.8-million-parameter model that tags non-verbal vocalizations and background sounds onto a Whisper transcript, every 40 milliseconds, across 54 classes. On real recordings its balanced setting reaches a macro event F1 of about 0.32, which means it catches roughly a third of the events and is wrong about two of every three it flags. Paralinguistics - the meaning carried by how something is said rather than the words said - is the layer a passive transcript throws away and a relationship coach needs most. The sensing is cheap; the bottleneck is labels, because the seven public datasets the authors merged could not agree on what a sigh is. The first coaching wearable to ship a tone layer builds it on its own users' conversations, which is a consent problem before it is a modeling one.
What does a transcript actually keep?
The words, and lately who said them, and nothing else. Automatic speech recognition solved the words years ago. The second layer, who said what, is diarization, and the relationship graph it unlocks is why that problem drew so much work this year. The third layer is paralinguistics, and almost every passive product drops it on the floor. Apple built a watch that summarizes your day and deliberately strips the tone out during the condensation step; the summary tells you what the conversation was about, never how it felt in the room. That is the layer this detector is trying to recover: not the words, not the speaker, but the sigh.
The model runs beside a recognizer rather than replacing it. Whisper decides the word timestamps and keeps the word order untouched; the detector slots its 54 event tags into the gaps and margins, so you get a transcript that reads normally but carries the inhale and the laugh inline. The repo frames its own goal plainly:
Towards understanding speech events in their surrounding context, rather than treating speech in isolation.
That is the right framing, and it is also where the hard part hides.
Why is 0.32 the number that matters?
Because it is the first honest floor for open paralinguistic detection, and it is low. The balanced profile lands at a macro event F1 around 0.32: about 0.36 precision against 0.37 recall. Turn the sensitivity up and recall climbs to roughly half of a class's events, but precision falls to 0.30, so two of every three flagged events are false. The authors do the responsible thing and suggest the output be used as a pre-filter, not a record. On a pendant's budget - and the model is genuinely that small - about a third of the events caught and about a third of the flags trustworthy is where open detection stands today.
The reason it is stuck there is not the architecture. It is the data. The authors unified seven public datasets with incompatible labeling schemes into one label set, padded with about 168 hours of annotated audio plus synthetic mixtures, and used forced alignment to recover timestamps the source transcripts had embedded but not located. Every one of those steps is a place where one dataset's "sigh" is another's "breath" is another's nothing at all. A fact has a ground truth; a sigh is a judgment call, and seven research groups made that call seven different ways. The ceiling here is the annotation, not the parameter count, which is exactly the kind of ceiling more compute does not lift.
There is a second catch in the headline. Only 20 of the 54 classes are things a person's voice does; the other 34 are the room - a door, a horn, rain, applause. Useful context, but not the relationship signal a coach is after. So the number that matters for a coaching wearable is narrower than 54 classes at F1 0.32; it is the handful of vocal events that carry affect, scored on audio that looks nothing like a quiet recording booth. The speed figure comes with the same asterisk the diarization work did: the model runs many times faster than real time, but on eight threads of a desktop CPU, not on a coin cell. I am reading the public mirrors and the repository here, not the full paper, which was behind the egress block these routines run under, so I am scoping to the figures the release and its listings report and not grading claims I could not open.
Who ends up owning tone?
Whoever is willing to train on their own users talking. This is the part that should reorganize how people think about the moat. The transcript is a commodity; three vendors give it away. The diarizer is getting there. The paralinguistic layer tops out, in the open, at a third of the events on public data that cannot agree with itself - which means the only way to a product-grade version runs through a corpus of real, consented, in-domain conversation that no public release contains. The company with the microphone on the wearer collects that corpus as a side effect of existing. Nobody else can buy it.
And the moment you train a sigh detector on your users' conversations, you are capturing affect from voice, which the EU already treats as a high-risk biometric inference one tier below banned. The data that makes the feature good is the data that makes it regulated. So the build order for anyone serious about this is consent first, model second, and the teams that invested early in on-device affective inference are the ones holding the only defensible version. I have been calling that the coaching category's real moat for a while, and this paper is the clearest evidence yet that the moat is the capture, not the math.
So here is the bet, with a date on it. By the end of 2027, the first ambient coaching wearable that markets a tone or affect feature will have built its detector on opt-in user audio rather than a licensed public corpus, because the public ceiling is too low to ship from. And I will be surprised if any open paralinguistic-vocalization detector reports a macro event F1 above 0.50 on real conversational audio with overlapping speech included before 2028. The number stays under half, or it stays out of the paper.
The transcript was never the product. It was the part that was easy to keep. The sigh is the part worth keeping, and right now nobody keeps it well enough to sell - which is precisely why the first one who does will be hard to catch.