As far as I can determine, psychiatrist Carl Gustav Jung was the first clinician to study emotional undercurrents in the human soul by way of technology.
It is 1905. Jung works at the Burgholzli psychiatric hospital as head psychiatrist. Aware of an unconscious emotional process that constantly inform our common waking states, he decides to use Galvanic Skin Response (GSR, now called electrodermal responses) to attempt to measure excitation of the Autonomic Nervous System (ANS) as more than just background noise. There must be meaning in these fluctuations of the ANS. By giving the test subjects prompt words, and measuring, among other things, the time lapse between the stimulus word, the subjects response, and the GSR activity, one of the oldest measurements of ANS activity, he finds that images likeFather, or Mother bring up ANS background noise of a very complex nature. In the background of consciousness, he identifies complex networks of disparate emotions interacting simultaneously. These interactions are much more than just the basic emotions. They are a complex mixture of basic emotional elements pervaded by a consistent feeling tone, of far greater meaning than the sum of the basic emotions contained therein. These volatile emotional background mixtures he calls‘Complexes’ and names his brand of psychoanalysis ‘Complex Psychology’. When I started my training at the C.G. Jung Institute in Zurich in 1971, that was still the name of Jung’s psychology, explicit in the German subtitle of theInstitute, while in English it had come to be called Analytical Psychology.
While Jung only measured the amplitudes of the GSR graph, it was my hypothesis that, more important than just the spikes, it was the wave forms over time that would display a meaningful signature graph of ANS arousal patterns which, from the emotion onset to its completion, represent an interpretable emotional phrase.
In 1995 I began researching this hypothesis with the generous help of Bruce Mehler at NeuroDyne Medical, a company leading the creation of electrodermal hardware. In 1997 we continued the research at the MIT Media Lab.
Let me explain:
Emotions are like music. They play out over time. Therefore, I’m using language of music theory to describe emotions.
emotion reverberates through the body from onset to resolution when it echoes away. Following musical theory as metaphor for emotion recognition I call this an emotional phrase.A phrase is a unit of musical meter that has a complete musical sense of its own. In Western musical theory, a cadence (from Latin cadentia 'a falling') is the end of a phrase in which the melody or harmony creates a sense of full or partial resolution. The atmospheric quality evoked by a complex emotional pattern I call, following C.G. Jung, the feeling-tone.
Since a single note has no intrinsic meaning, as it is just a building block of a meaningful musical or emotional phrase, I felt that individual spikes in ANS activity matter less than the phrase.
Emotions are analog events. The measurements of electrodermal responses are analogues of autonomic nervous system (ANS)activity. A change in the electrical resistance of the skin is caused by a physiochemical response to emotional arousal of the ANS, thereby making sympathetic nervous system activity readable. These measurable skin responses are an analogue of the arousal of the ANS, which gives rise to the sweat response. The amount of sweat on the skin determines the conductivity of the skin which can be measured.
In music recording, a device called an analog-to-digital converter (ADC) takes the electrical energy and converts it to a digital file for storage. The ADC samples the frequency and amplitude at a set rate.
If we want to digitize emotions, we must sample them at set intervals to determine their amplitude. We did that through sampling of galvanic skin responses. Each sampling unit, like a particular spike, corresponds to an individual note. When looking at the amplitude samples over time we notice forms on the graph. These forms we call signatures. These signatures combine into a phrase that ends in a cadence. Such a phrase corresponds to a complex of emotions that together form that phrase. Such complexes are mixtures of basic emotions pervaded by a recognizable feeling tone and organize around an emotionally charged image. C.G. Jung redux.
Basic emotions are abstractions. In lived reality, emotions present themselves as complex mixtures. Therefore, the technology of emotion recognition should be based on complex emotions, not basic emotions.
At Attune Media Labs we have developed and patented ways of measuring complex emotions in real time and use them for the human/GenAI interface, thereby developing true GenerativeEmotional Intelligences (GenEI or genei) that interact with the user.
The process is as follows:
Our virtual human agent we call Mim, because it mimics human consciousness, continuously measures arousal in the user. Currently, we do this by way of vocal stress qualities. Over time Mim finds particular emotional signatures playing out over around 20 second periods. When Mim finds such comparable signature patterns it asks the user: “When you just spoke, I noticed the same feeling as when you talked about xxx last week. What’s that all about?” In the conversation that ensues an emotional language develops between user and Mim that is very similar to the shorthand developed in any longer-term meaningful relationship. This emotional shorthand makes it possible to create a salient emotional memory over time which facilitates the preconditions for truly intimate connections that successfully mimic actual inter-human relations between users and generative emotional intelligences or genei, pronounced as ‘genii’,the plural of ‘genius’.
