The AI that will be able to read your emotions
While we communicate with words, an important part of communication takes place non-verbally. But how do we read body language and reflect each other in a time when more and more meetings take place digitally? In a new research project, researchers are developing an AI system that can read emotions – which can, for example, be used as an advisory tool for therapists.
"Communicating with other people is an extremely complex process. We are both senders and receivers of verbal messages - at the same time we communicate non-verbally, simultaneously transmitting and receiving signals on both conscious and unconscious levels," says Lennart Högman, researcher at the Department of Psychology at Stockholm University and one of the researchers behind the project "Let's talk about non-verbal communication."
The research team is looking at different forms of verbal and non-verbal communication, including facial expressions, body movements, approach or avoidance behaviors and tone of voice to name just a few.
"The face conveys a lot of information. We have at least 40 muscles in the face which allow us to communicate with each other in very elaborate ways, enabling us to solve things effectively in a group. Having coherence in communication - i.e. knowing whether we are being mirrored or not, is very significant for us," says Lennart Högman.
Serves as advisory tool to therapists
This has, however, become much more difficult as more and more meetings are held digitally. Lennart explains that about 96 percent of all therapists today offer online therapy in the USA, compared to just over 20 percent in 2019. The numbers are thought to be similar in Sweden. How are we affected by meeting online so much? How do we mirror each other in these settings and what are the differences compared to meeting in real life? These are questions which the researchers behind the project hope to learn much more about.
The main purpose of this research is to develop an AI system that will serve as an advisory tool to psychotherapists. This AI system should be particularly useful for those practicing internet based therapy, but could also be useful for therapy where people meet face-to-face. The system willserve to primarily complement dynamically oriented therapies, such as psychodynamic therapy, but should also be a helpful tool for CBT and schema therapy, which is used, for example, in treating emotionally unstable personality disorder (EUPD). Lennart Högman believes that the tool could also be used in couple therapy, although this may be a bit more complex.
Following emotional development over time
One of the goals of the project is to investigate whether treatment with internet therapy can have the same beneficial long-term effects as therapy conducted face-to-face.
"It is difficult to sense other people's feelings when conveying information via internet, especially because you do not get the same eye contact. It creates completely different conditions for interaction. Can you compensate for this? Do you, for example, switch to listening more to the voice?"
The tool makes it possible to follow a client's emotional development over time. It is also possible to see when the therapist and client simultaneously exhibit certain types of feelings and when differences occur in this synchronization. Importantly, researchers believe that this form of synchronization is important for treatment outcomes. It is relatively common for clients to terminate therapy prematurely, something which Lennart Högman believes may have to do with some clients not feeling that they are getting the contact with the therapist that they expected.
Captures conflicting signals
The AI tool will also provide detailed summaries of interactions, and may, in the future, also be able to provide answers to specific questions in a way which is similar to systems like ChatGPT.
"For example 'can you summarize everything this person has said about their mother, their father, a place or whatever', whilst also providing information on the positive and negative emotions that have been shown in different contexts. The system becomes like an assistant who remembers what has happened during the entire therapy," says Lennart Högman.
In this way, it can serve as a tool to track and measure quality and improve treatment. At the same time, he fully understands that not everyone will want to use the tool.
"If you have been working as a therapist for 30 years, we understand and we respect that you may not want to use it. But in the near future it could be very useful. It is easy to overestimate your own knowledge of how other people think. We use own way of functioning and apply it to others and we also have limited memory resources”.
The tool should also be able to show when a client conveys conflicting signals, highlighting when what a person says does not match their body language. But Lennart Högman is careful to point out that their technology should not be used in any kind of interrogation situations.
A measure of emotional competence
Besides therapy, Lennart Högman sees a number of other potential applications for this new tool. For example, in educational contexts. Not only in training future therapists, but also in supporting other types of training where it is important to measure how people react emotionally in different contexts.
"The problem has previously been measuring emotional intelligence (EQ) or emotional competence. Most measures of EQ are based on self-assessments or other simple tests. This could be a very good objective tool for measuring emotional competence, to determine if a person has difficulties with interaction and in what way it manifests itself."
Another context could be in recruitment, although this may not be entirely unproblematic, according to Lennart Högman. The anxiety most people feel before a job interview would of course be exacerbated if they were also being judged by an AI. Other potential arenas, such as online dating apps may also benefit from using this technology, although this is not currently included in the project.
Important to try to avoid prejudice
A problem with AI that has been much discussed is the risk of recreating prejudices.
"It's always for better or worse - there are no AI systems that are unbiased, they are trained on data created by humans and have their built-in prejudices."
The important thing is to try to build a system that avoids prejudices as much as possible, and to be aware that you constantly need to make improvements. It is important to have training data that is very broad and selected in such a way that you should avoid prejudice while developing the system to follow all ethical guidelines, says Lennart Högman.
"And the idea is that it should be an advisory instrument, not that the AI should makedecisions on its own."
At the same time, it is advantageous that the AI system will not be guided by memory images in the same way as humans often do following a conversation. Here everything is collected in black and white and transcribed, in the summary you can also see where in the conversation the people have acted emotionally. However, it is the therapist who is responsible for the interpretation, says Lennart Högman.
"Human reactions are complicated, there is a tremendous amount of information in a meeting between two people. One could say that through the project we are trying to open the doors to the unconscious processes in interpersonal encounters," concludes Lennart Högman.
About the project
The project, which has received grants from the Marcus and Amalia Wallenberg Memorial Fund, is a collaboration between the Department of Psychology and the Department of Data and Systems Science as well as a more informal collaboration with the Department of Media Studies and international researchers.
The tool is developed in collaboration with the company Empatik.AI, which is financially supported by Stockholm University. It contains several different layers of AI systems that, among other things, analyze emotional reactions via video and sound. An initial level analyzes motor activity, movement patterns for face and body, parallel to sound - so-called emotional prosody (what we express with the voice). Then the entire conversation is automatically transcribed. Finally, the different data flows are combined with the help of AI. No cloud services will be used due to the extreme sensitivity of the data
Last updated: July 12, 2023
Source: The Communications Office