No matter how debonair you are at your best, conversation can be awkward for anyone. That’s especially true for those who struggle to pick up on social cues. To help navigate those rocky exchanges, MIT CSAIL researchers have created a wearable system that can tell whether the person you’re talking to is happy or sad. It’s a start.
The device takes an existing research-grade wearable—Samsung’s Simband smartwatch, which can measure movement, heart rate, blood pressure, blood flow, and skin temperature—and pairs it with audio capture that can pick up signals like tone, pitch, energy, and word choice, and provide a transcript of the text. By weighing all of the incoming signals, algorithms can classify each five-second installment of conversation as either “positive” or “negative.”
“You have a GPS in your pocket, it’s very complicated technology,” says study co-author Tuka Alhanai. “But we don’t have a GPS for social interactions.”
Signals and Noise
The team started with over 500 signals that could potentially tip off how a conversation was going—ranging from movement, to speech patterns, to individual word choice—and let on-board artificial intelligence sort out which were most important, rather than letting preconceptions dictate.
“With very little structure the algorithm was able to arrive at what we intuitively thought,” says study co-author Mohammad Ghassemi. Long pauses are more likely to come during sad stories, for instance, as well as fidgeting. Subjects told happy tales with more varied speech patterns. Hearing someone isn’t as effective as being able to see them as well.
It may sound obvious to some, but these are also the types of social cues that can be difficult for those with anxiety, or for those on the autism spectrum. Having a device that can take the temperature of a room could go a long way to mitigate those issues.
That’s also what makes the CSAIL study’s form factor so critical. There are other systems designed to translate tone into intent, but Ghassemi notes they’re often useful only in tightly controlled lab settings, or they require specialized equipment to work. The tricked out Simband looks like any other smartwatch. It’s just a little smarter, in this very specific way.