In this video we will learn about Gesture Referentiality, which is one aspect of the meaning dimension of gesture. We’ll start with defining gesture referentiality, then, we will discuss the different types of referential meaning that a gesture may convey while looking at some examples. Lastly, we will learn how to annotate gesture referentiality in ELAN.

Previous gesture classification systems have tried to define different gesture types on either how they represent meaningful content in speech, such as McNeill’s iconic, metaphoric or deictic gesture types, or how they associate with speech prosody to convey discourse-pragmatic meaning, such as McNeill’s beat gesture type. As we mentioned in the video introducing M3D, the M3D approach differs by assessing how a gesture conveys meaning in a non-mutually exclusive way, independent of its association with  speech prosody. In other words, a single gesture may or may not convey referential meaning; may or may not convey discourse-pragmatic meaning, and may or may not associate with prosodic prominence.

Gesture Referentiality refers to how the meaning of gesture relates to speech content. In terms of Gesture Referentiality, M3D proposes three categories of gestures: emblems, referential gestures, and non-referential gestures.

Emblems

The first category, emblems, refers to gestures that are culturally-defined and have direct form-meaning connections, such as the peace sign. We can do the peace sign without speaking and people will generally understand this gesture if their culture shares this emblem. If the form of the peace sign is slightly altered, suddenly we are no longer doing the peace sign. For example, if we bend a finger, this is no longer recognized as the peace sign. As I mentioned before, not only is the form of the gesture very specific, but it must have a particular meaning shared by a culture. Indeed, a single gesture may have different meanings depending on the culture in which it is used. For example, if we lift our thumb together with our little finger, it means relax in Hawaii, while in China, the same gesture is used to represent the number six.

Referential Gestures

The second category is referential gestures. Gestures are considered referential if the form of the gesture pictorially represents, or indicates spatial relations of content in speech. For example, I can say “Mary caught the ball” and I can use my hands to pictorially represent the ball, indicating the size, shape, etc. This gesture illustrates a specific, concrete object that is expressed in speech. Therefore, we could say that this gesture conveys meaning through iconicity. Now, imagine we weren’t talking about a ball, but we would be talking about a more abstract concept like marriage, saying something such as “Mary and Steve got married.” The same gesture would be representing the metaphor of a union via the joining of the two hands. Such a gesture would convey meaning through metaphoricity. Finally, we can use gestures to refer to space, and thus convey meaning through deixis. For example, if Mary and Steve are in the room, I can point at them when I say “Mary and Steve got married”. This would be concrete deixis, in that, you are pointing to an object present in the environment. But imagine if I’m telling a story about Mary and Steve, how they met, I can locate these characters in space and point to them as I say things like “Steve caught Mary’s ball, he went over to her, and it was love at first sight”. This would be considered abstract deixis.

Importantly, multiple referential meanings can be conveyed in the same gesture simultaneously. Here are some examples. In this video clip you can see a gesture conveying both iconicity and deixis.

[see example]. It is iconic because she is talking about something reaching the public, and her hands move towards the public visually illustrating “reaching the public”. Additionally, she performs the gesture to indicate the location of the audience in front of her. 

Here is another example showing dimensions of both metaphoricity and abstract deixis [see example].

As she says “and get research grants down the road” she is forming her hands in such a way to visually represent a road or path, which is used as a metaphor to talk about the future. Additionally, she performs this gesture with a forward motion, indicating that “down the road”, or the future, is in front of her.

Non-referential

The third and final category is non-referential gestures. These gestures do not represent speech content via iconicity, metaphoricity, or deixis. That means, they do not visually illustrate content in speech, or use space in a meaningful way relative to the utterance. Non-referential gestures may include movements like McNeill’s “beat” gestures. It can also include other movements like “hand flips”. Even though these gestures do not convey referential meaning, they are most certainly not devoid of meaning. In fact, they may convey more discourse-pragmatic meaning. For example I could say something like “Mary caught the ball”. In this case, the gesture does not convey meaning through illustrating or pointing to Mary. However, its co-occurrence with Mary in speech may have important pragmatic functions, such as marking new or contrastive information in speech.

Annotating in ELAN

Now, how do we annotate the referentiality in ELAN? Unlike labeling for the form or prosodic dimensions, the aspects of the meaning dimension are always annotated with the sound turned on. We want to base our referentiality annotations on how the gesture represents what is being said. To better understand the context, we play a large chunk of the video around the gesture of interest – do not only play the stroke phase.

First, consider if the gesture stroke is an emblem or not. If so, annotate the span of the stroke as such in the “Emblem Tier”. If not, check to see if it is conveying any aspects of the discourse content, which may include iconicity, metaphoricity, deixis, or any combination thereof. In the ELAN template, you will see a tier for each sub-dimension, specifically “iconicity”, “metaphoricity”, and “deixis”. Annotate any and all referential meanings you clearly see being conveyed in the gesture on these tiers. So for example if you see a gesture portraying metaphoricity and deixis, you would create two annotations, one on each of those tiers for the duration of the stroke. Finally, if you fail to clearly see any referential meaning, then label the gesture as non-referential by adding an annotation on the “non_referential” tier. 

In this video, we’ve explored what gesture referentiality is and how we annotate it. Remember: Gestures can be emblems, which are culturally-specific gestures or they can be referential, if they portray visual or spatial content in the utterance, or they can be non-referential, if they do not visually or spatially refer to content in the utterance. Importantly, when gestures are referential, the meaning can be composed of a combination of different referential dimensions, including iconicity, metaphoricity, and deixis. Thanks for watching!