This AI Wearable Helps Make You Less Socially Awkward

For the socially awkward among us, interacting with other people can be a painful experience, but for those with social anxiety disorder or Asperger's, it can seem downright impossible. Fortunately, a new wearable that is able to determine the mood of a conversation based on speech patterns, tone of voice, and physiological information like a person's heart rate, blood pressure, skin temperature, or hand movements, might help alleviate the stress of human communication.

"The way we set up our experiment was to explore if we had information on half of a conversation, could we reconstruct how they felt as they were telling these stories," Mohammad Ghassemi, a graduate student at MIT's Computer Science and Artificial Intelligence Lab (CSAIL), told Motherboard. 

Ghassemi and fellow CSAIL graduate student Tuka Alhanai recorded the vitals and audio from stories told by 31 students at MIT, who then self-reported on whether they considered the overall story they told to be happy or sad. A lab technician would then break each story down into five second segments and label each segment as either happy, sad, or neutral.

Some of these stories were then used to train two different neural networks—a type of AI architecture that is modeled on the human brain—to classify the overall mood of the stories and the mood of the 5-second segments of these stories, respectively. Based on the AI's analysis, the researchers found that certain patterns of human communication can be clear indicators of the mood of a story. For instance, sad stories are much more likely to be monotone, include long pauses, fidgeting, and have the storyteller put their hands on their face. Still, this didn't guarantee that the AI would be able to understand the narrative's mood.

"There's a lot of variation in the way we tell stories," said Ghassemi. "A sad story can have happy moments, or a happy story can be sad up until the very end. This variation can make the emotive content of stories hard to classify."

Based on these observations, the AI was able to capture the overall mood of a story with 83 percent accuracy. The neural net's ability to classify the short segments was only 17.9 percent better than if the machine had randomly guessed. According to Ghassemi, this is due the difficulty of trying to analyze very small segments of conversation without reference to the whole conversation, coupled with the relatively small data set used in the initial research (stories from 31 students).

As the CSAIL researchers collect more data, not only will the AI be able to better understand the mood of parts of a story, but it will also add far more "emotional granularity." So rather than just telling the user if a story is happy or sad, it can measure more nuanced emotions like boredom or fear, as well as how the emotion of a speaker changes over time.

Given how much data is being collected by the wearable, Alhanai and Ghassemi have spent a lot of time figuring out how to tackle the privacy issues raised by its use. Obviously, the wearable requires consent from the person who will be wearing it, but the researchers also wanted to make sure that the collected information will never be available to anyone but the user. As such, all the recorded information is stored locally on a user's phone and the device is not internet connected, which would make it much more vulnerable to exploitation.

Although Alhanai and Ghassemi have a lot more testing to do before their AI device is ready for the market, they hope that it will make communication easier for people with social anxiety disorder or Asperger's once it does leave the lab. Not only would this allow the user to go back and review a conversation after the fact to see how it went, but Ghassemi also envisions a future in which the device is able to tell users in real-time how the conversation is going if two people in a conversation are wearing the device. For example, the wearable might give two quick vibrations to the storyteller if the vitals of the other person indicate that they are beginning to get bored by the story.

"The next step is putting a wearable on both parties to see more interesting sorts of things like how one person's communication might affect the other person's communication," said Ghassemi. "This research is a gateway to put more significant research efforts on human communication."

from This AI Wearable Helps Make You Less Socially Awkward