Psychologist Dr. Paul Ekman summarized his decades of ground-breaking research on nonverbal communication in his 2016 book, Nonverbal Messages: Cracking the Code. He spent years demonstrating that basic human emotions are displayed with the same facial expressions literally around the world, and his approach to cataloging individual expressions that can reinforce or contradict what a person is saying is used by law enforcement and security personnel to detect deception as well as by those who want to improve interpersonal relations.
Ekman takes an empirical approach to his work. For example, in developing his facial expression classification scheme (the Facial Action Coding Scheme or FACS) he identified each muscle in the human face and then took pictures of what faces look like as each muscle is contracted. When testing the ability of observers to use facial and other nonverbal cues to detect deception, he would film people who were either lying or telling the truth and use the films to test the ability of multiple sets of people to detect the deception.
There are some interesting parallels between Dr. Ekman’s life work and the tasks faced by those who manage unstructured content:
1. Spoken or written words are only part of the story. In behavioral science, words spoken by subjects are only part of the story. Nonverbal acts like gestures or facial expressions can completely change the spoken word. In fact, some important communication interactions may not have any spoken words, e.g., hugs, fist fights or hand gestures. In content management, the words that are available for automated analysis may be inadequate to classify or convey the meaning intended in document by their authors. Page size, orientation, layout, and non-textual graphical elements like logos, lines, pictures, and illustrations can be very significant in discerning the meaning of enterprise content. Some layouts have general meaning (e.g., a letter), some have industry meanings (e.g., well logs), and some are company specific. And like nonverbal interactions, some files or documents may have no words available for analysis.
Ekman thought that the standard psychoanalysis practice of having psychoanalysts sit behind patients deprived analysts of the insight that direct observation of nonverbal communications would provide. In the enterprise content management world, using just words or just text is like having psychoanalysts just read real-time transcripts of what patients are saying.
2. Classification starts by objectively defining basic elements. Ekman went to considerable lengths in developing his facial expression classification scheme to define each observable component of facial movement called an Action Unit (“AU”) describe how to decompose each facial expression into their constituent AUs. He tests objectivity by having multiple people rank or score the same expressions. Without objectivity and replicability, any system is subjective and inherently unscalable. In content management, the basic unit being managed is the document. Content managers should maintain a standard set of files or documents that are periodically reprocessed to be sure than in developing or tuning their automated classification system they have stopped yielding consistent results. Inconsistent results become unusable at some point causing a major loss of return on the investment in the system.
3. Classification is ongoing. In a major article published in 1969, “The Repertoire of Nonverbal Behavior: Categories, Origins, Usage, and Coding,” Ekman and co-author Wallace Friesen identified five types of nonverbal communication including facial expressions and gestures, and suggested a framework for analyzing them using several factors such as:
- Whether the act was intentional
- Ability of observers to determine meaning (idiosyncratic if only one person would know the meaning, shared if a set of observers would know)
- Relationship of the acts to the words spoken.
Despite having worked in this field for many years, they noted that their classification scheme was ongoing. The same is true for enterprise content classification schemes – content is constantly evolving and classification schemes must evolve to keep pace with the content.
4. Use multiple viewpoints to guard against bias. In his book, Ekman pointed out that he always tries to guard against having bias influence the process or results by having people on the team who differ in their approach to the subject of the study. Content managers should include experts whose experience goes beyond the traditional text-based approach to managing content.
5. Overconfidence in accuracy of results may be misplaced. One interesting insight from Ekman’s work in testing the consistency of results was that police officers and social workers were more confident in their ability to detect lies but they scored no better than other groups. To be sure that confidence in consistency of their classification system is not misplaced, content managers should consider having a consistency audit performed by a third party.
For more information on how visual but non-textual information can be used to consistently classify unstructured documents and files, you can download my book, Guide to Managing Unstructured Content, for personal use for free at: https://beyondrecognition.net/download-john-martins-guide-to-managing-unstructured-content/
Paul Ekman, Ph.D., Nonverbal Messages: Cracking the Code, My Life’s Work, published in 2016, https://www.amazon.com/Nonverbal-Messages-Cracking-LIfes-Pursuit-ebook/dp/B01M8LDPQX/
Paul Ekman and Wallace V. Friesen, “The Repertoire of Nonverbal Behavior: Categories, Origins, usage, and Coding,” Semiotica, 1969, I (I), 49-98. http://homes.di.unimi.it/~boccignone/GiuseppeBoccignone_webpage/CompAff2011_files/EkmanFriesenSemiotica.pdf
Wikipedia page on Ekman: https://en.wikipedia.org/wiki/Paul_Ekman
Ekman’s current website with links to various courses in using emotion recognition: www.paulekman.com