#Tokyo Denki University | Japan 🇯🇵

[xtruss]

Communication: Your 🧠 On Emoji. It's Complicated. And That's Good.

— By Alla Katsnelson | Nautilus

Image: eztexting

Twenty years ago, Microsoft’s instant messaging platform added a new feature: dozens of little icons users could drop into their messages, conveying happiness, surprise, confusion, or a sheep. Gradually, then all at once, emojis were here: spreading from chat platforms to SMS, email, social media, and—to the chagrin of legions of teachers—even infecting school assignments.

For years, I was an emoji hold-out. Embracing the little, cartoonish images felt like transgressing against the virtue of words. To my linguistically traditional soul, raised on Jane Austen and Isaac Babel, emojis seemed cheap and unnecessarily revealing. I resented their creep into written communication, which had long managed just fine, thank you very much, with the alphabet.

But just a few years ago, after befriending a colleague whose texts were flecked with these symbols, I had a change of heart. Our daily banter thrived on the emotional zest that emojis added, and on the sense of connection they fueled. Timidly at first, I started to thread them into my digital discourse. Now they’re woven into my communication with many people in my life, punctuating a short note or standing alone as a single message, a 💥 or 🔥 or 🌟 as a full-stop reply. What’s surprised me most is the palpable joy these flutters of icon-based interaction have added to routine exchanges.

The Effect is Like a Shot of Meaning-Making Caffeine—Pure Emotional Charge.

Valeria Pfeifer is a cognitive scientist at the University of Arizona. She is one of a small group of researchers who has studied how emojis affect our thinking. She tells me that my newfound joy makes sense. Emojis “convey this additional complex layer of meaning that words just don’t really seem to get at,” she says. Many a word nerd has fretted that emojis are making us—and our communication—dumber. But Pfeifer and other cognitive scientists and linguists are beginning to explain what makes them special.

In a book called The Emoji Code, British cognitive linguist Vyvyan Evans describes emojis as “incontrovertibly the world’s first truly universal communication.” That might seem like a tall claim for an ever-expanding set of symbols whose meanings can be fickle. But language evolves, and these ideograms have become the lingua franca of digital communication.

The story of the emoji reaches back further than early instant messaging programs. Before these graphically detailed icons were easy to display, the clunkier, character-constructed emoticon held their place.

Carnegie Mellon computer scientist Scott Fahlman is often credited with codifying these smile-and-wink punctuation constructions. After watching posters in early online bulletin boards get into skirmishes—say, when a poster’s sarcasm was misread—he suggested in 1982 that colleagues add a ":-)" or ":-(" to indicate their tone. If posters could flag when they were being funny or sarcastic, he figured, readers wouldn’t be so easily upset.

Writers and thinkers had, for decades, proposed subbing in punctuation for feelings, though many, it seems, did so in tongue-in-cheek jest. Other early potential emoticons in the wild—such as a ";)" in a transcript note describing audience reactions of “applause and laughter” during an 1862 speech by Abraham Lincoln—were likely typesetting errors or examples of looser punctuation norms of the day.

The interface of language and emotion is where the magic lies.

None of these uses took root though until the fertile conditions of the early internet arrived. And as graphical interfaces improved, the contemporary emoji was born.

The emoji didn’t initially set out to be a souped-up emoticon. When, in 1999, Japanese artist Shigetaka Kurita developed a first suite of 176 of them for the cell phone company he was working for, most weren’t meant to convey a feeling at all. The majority of them were quotidian symbols he envisioned people would toss in occasionally in place of words: a house, an ear, a tennis racket, a fax sign.

It wasn’t until 2011, when Apple first made emojis accessible through a dedicated emoji keyboard on their mobile devices (and Android did the same two years later) that emojis truly started going mainstream. By 2015, more than 90 percent of internet users had deployed them, and the Oxford English Dictionary named 😂 Word of the Year. Today, the Unicode Consortium, emojis’ governing body, as it were, lists more than 3,500 of them.

The word emoji itself has nothing to do, etymologically, with emoting. It’s a blend of the Japanese words for picture (e) and character (moji)—unlike emoticon, which is an American mix of emotion and icon. This difference in origin and intention also inflected early scientific research into these new communication tools and their impact on the people using them.

Perhaps the first study of how these visual representations activate the brain was presented at a conference in 2006. Computer scientist Masahide Yuasa, then at Tokyo Denki University in Japan, and his colleagues wanted to see whether our noggins interpret abstract symbolic representations of faces—emoticons made of punctuation marks—in the same way as photographic images of them. They popped several college students into a brain scanning machine (they used functional magnetic resonance imaging, or fMRI) and showed them realistic images of happy and sad faces, as well as scrambled versions of these pictures. They also showed them happy and sad emoticons, along with short random collections of punctuation.

The photos lit up a brain region associated with faces. The emoticons didn’t. But they did activate a different area thought to be involved in deciding whether something is emotionally negative or positive. The group’s later work, published in 2011, extended this finding, reporting that emoticons at the end of a sentence made verbal and nonverbal areas of the brain respond more enthusiastically to written text. “Just as prosody enriches vocal expressions,” the researchers wrote in their earlier paper, the emoticons seemed to be layering on more meaning and impact. The effect is like a shot of meaning-making caffeine—pure emotional charge.

It was surprising that these punctuation faces carried their emotional valence to the reader’s psyche without first being recognized as abstract faces. Back then, many researchers assumed people first pieced together the line-and-dot faces and then inferred their expression “as a bottom-up process,” Yuasa explained to me via email. But the results suggested that the emoticons were plugging into something more foundational even than face-recognition—hinting that responding to emotion in communication is a primal, even deeper drive.

A few years later, researchers in Australia reported that people were much quicker to grok smiley face emoticons ":-)" as faces than when the same symbols were typed backward: ")-:" For the lead researcher, Owen Churches, the results pointed to our brains’ amazing ability to adapt to a quickly changing world. “There is no innate neural response to emoticons that babies are born with,” he told ABC Australia. “This is an entirely culturally created neural response.”

Once the visually richer emojis proliferated, scientists had even more concepts they could interrogate to discern the real-time integration with language, communication, and feeling. And the research quickly became intriguingly nuanced. To wit: Do emojis and words have similar functions when attempting to convey irony? Irony in its most basic form is expressing the opposite of what you mean, to make a point. The incongruity it conveys is so cognitively satisfying precisely because of the layers of drama and meaning it adds to language.

Benjamin Weissman and Darren Tanner at the University of Illinois recorded patterns of brain activity as participants read simple sentences ending with different face emojis—one that aligned with the meaning of a sentence, one that diverged, and one of a wink-face emoji clearly signaling irony.

Comparing their findings to previous research on how the brain responds to ironic language, they reported—in their 2018 cheekily titled paper, “A strong wink between verbal and emoji-based irony”—that, as far as the brain is concerned, emojis and words do roughly the same job.

“There’s basically a match,” says Weissman, a cognitive scientist now at Rensselaer Polytechnic Institute in Troy, New York. “As long as there is some sort of irony conveyed and interpreted, the brain response looks pretty similar to the brain response for traditional non-emoji irony.” That finding aligns with more recent, still-unpublished work Weissman did with cognitive scientist Neil Cohn at Tilburg University in the Netherlands. There, they probed how the brain responds to reading sentences in which either a word or an emoji matched the expected meaning or had an unexpected meaning:

Here, too, the brain responded to the emojis pretty similarly as to the words, with the expected ones eliciting a brain activity pattern associated with linguistic prediction, and the surprise ones yielding activity associated with processing mismatched meaning.

In a way, Weissman says, it doesn’t really matter whether we are calling to mind a concept from a word or an icon. On the whole, for higher-level cognition in which the brain is making complex meaning from inputs it receives, it can integrate all sorts of elements, including facial expressions and tone of voice, he says. And emojis are just another type of this input. “The meaning-making process can probably operate on a level independent of the modality itself,” he says.

But of course, anyone who has wielded emojis knows they are doing something more than just colorfully, jauntily standing in for words. Like their 41-year-old cousins, the emoticons, they are doing heavier lifting, too. Their ability to convey emotion does a complex dance with language. For Pfeifer, this interface is where their magic lies.

“Emojis have this ability to make the same words seem more emotional or less emotional—to seem alarming versus completely fine or joking even,” she says. These are important social functions, she adds. “Through our switch to using a more text-based communication we lost this additional layer of meaning that emojis can now provide.”

Our species and our many spoken languages emerged in the frothy cauldron of in-person communication, steeped in tone of voice, volume, facial expressions, gestures, posture, knowing glances. Even as writing forms began to emerge, such as cuneiform more than 5,000 years ago, for the majority of history their use remained largely official—government, business, religious. Interpersonal, social, cohering communication was still in-person, and often by necessity; as recently as 1960, less than half of the global population could read and write, and even in Austen’s era of the seemingly ubiquitous romantic epistle, only about half of English people older than 14 years could have penned or read a letter.

Positive Emojis Say, ‘Hey, I’m Listening.’ Negative Ones Have a Very Different Effect.

But as we stepped into the brave new digital world, the written word has taken over much of daily social and collegial correspondence. Texts and direct messages rather than social calls or phone calls. And work meetings or calls now often transmuted into dashed-off lines of keystrokes on Slack or Microsoft’s Teams.

As they say, nature abhors a vacuum, and emojis seem to have arrived at a time when a new communication niche needed filling. “The way we communicated when we all started texting or emailing seemed to be deprived of something that emojis seem to fill,” Pfeifer says.

She and her colleagues were also interested in the ways in which different emotion-evoking emojis impact social dynamics in their coloring of written statements. They found that happy emojis, such as hearts and smiling faces, added a general emotionally positive boost to a message, though not in any terribly specific capacity. Instead, these pictorial cues served more as bids of connection.

“Positive emojis are like a blinking light on a recording device,” she says. “Maybe we send this type of emoji to say, ‘hey, I’m listening,’ or ‘I’m interested in what you’re saying’—just as a way to confirm the social relationship between us,” she says. These sorts of emojis seem to be fostering social cohesion.

Negative emojis, on the other hand, affected words and interpretation differently. Recipients in their study read frowns, angry faces, and tears as indicators of a much more specific mental state, and they processed these symbols much more carefully and more in-depth, Pfeiffer says. “It is a lot more revealing from the sender’s perspective to send a negative emoji than a positive one.” And while positivity serves as a social cement holding us together—even, it appears, in emoji form—negative sentiments require plumbing the depths of a relationship or of a shared understanding to clarify their intent. And just as when we’re engaging in-person, they are more likely to spur miscommunication.

Emojis are also filling other social gaps born from our shift to a digital lifestyle. In her work, Linda Kaye, a psychologist at Edge Hill University in Ormskirk, England who is writing a book that synthesizes emoji research, has explored how these ubiquitous icons can reveal valuable clues to a person’s personality.

Interactions through social media platforms are often assumed to be opaque—and they do lack many markers that humans rely on to understand and make accurate judgements about each other. Kaye decided to test whether the way people use emojis online might help others get to know what they’re like. She and her colleagues asked a group of people with a presence on Facebook to complete a written personality questionnaire, and then they had another group look at screenshots of their profiles.

Looking at five major dimensions of personality, they found that the kinds of emojis people used in their profiles helped viewers assess two of them—open-mindedness and extraversion—with reasonably good accuracy. And while extraversion is pretty easy to judge in person, open-mindedness can be tough to gauge. “That tells us that when we are forming first impressions, actually online sometimes might give us more behavior to help us understand open-mindedness than offline targets,” Kaye says.

It turns out that a world freshly speckled with 🙂s is not really such a changed world after all. Nor as flat of a one as many might assume. “Ultimately, communication—the purpose of it and the way we do it—inherently doesn’t change all that much,” Kaye explains. “I would say it’s more about expanding our range.”

To me, as I continue on my newly emoji-strewn path, that’s an inspiring thought because it suggests we don’t need to poopoo such novel concoctions but can see them as a triumph of the dazzling adaptability of the human brain. And I will seek delight in the fact that our species can not only access conceptual and emotional language so rich as to craft resonant works of literature like Moby Dick—but also can crowd-source a full translation of it into emojis: 🐳.