In this video we will learn about one aspect of the meaning dimension of gestures, what we term the pragmatic domain. First, we will situate the pragmatic domain within the meaning dimension of gestures. Then, we will define the different pragmatic domains, while exploring a few examples together. Finally, we will describe the steps in how to annotate the pragmatic domain.

As we mentioned in the video introducing M3D, the M3D approach distinguishes the meaning conveyed by a gesture in two aspects. The first aspect refers to a gesture’s referentiality, that is, how the gesture relates to concurrent speech, for example, via iconicity, metaphoricity, deixis, or if it is non-referential in nature. Regardless of gesture referentiality, gestures may (or may not) also convey discourse-pragmatic meaning. 

It is important to remember that speech is multimodal in nature, and as such, aspects such as speech prosody and gesture can contribute to the pragmatic meaning behind a message. For example, I can say “John is coming to the party?”, and it is very clear from the text, my intonation, and my gesture that I am asking a question. But if my gesture changes, I can completely change the pragmatic meaning behind the question. For example, if I say “John is coming to the party?” - I am clearly expressing how I feel about his coming to the party, and am changing the message I am sending to my listener, perhaps warning that I will not be happy if he is present. Alternatively if I say “John is coming to the party?”, the listener would probably interpret that I am asking if he will be driving to the party, and respond with additional information, such as saying “yes, but he’ll be coming by train since his car is in the shop”. Given the relevance of gesture in communicating pragmatic meaning, the pragmatic domain of gesture is the second aspect of the meaning dimension which specifically assesses how a gesture may convey discourse-pragmatic meaning. Through a thorough review of the literature on the pragmatic functions of gesture, we have identified 5 pragmatic areas, or domains, in which gesture may function pragmatically. 

The 5 pragmatic domains identified in M3D

The first pragmatic domain is Speech Act marking. This refers to when a gesture has its own illocutionary force. Specifically, gestures aid in displaying the intention behind an utterance, ultimately to bring about what is to be accomplished by the speech act. The various speech acts that a gesture can reinforce include Directives, such as commands, requests, and invitation; Representatives such as assertions and claims; Expressives such as greetings, apologies, or congratulations; Commissives, where speakers are committed to future acts, such as promises, pledges, or vows; and finally, Declarations which change an entities status, such as blessings, firing, arrests, etc. For example, this clip shows a gesture that functions as marking a directive. Pay attention to the first gesture [see example].  The speaker is talking about how in her search for an answer, she was led to ancient Greece and ancient Rome. She then says to the audience “stay with me” and the gesture reinforces her imperative request that the audience remain engaged and tolerate a seemingly meandering section of her talk.  

The second pragmatic domain is Operational Marking. This specifically refers to any aspects of affirmation or negation in gesture. Some straight-forward examples would be shaking the head to say “yes” or “no”. However, we can also indicate more complex examples, such as “hand sweep” gestures. For example here is a gesture which indicates negation. [see example]. The speaker describes how a bowling ball will always follow the same path, and specifies that it doesn’t matter what temperature it is or what’s in his way. He expresses this negation both through head nods as well as his hand gestures.   

The third pragmatic domain is Stance marking. M3D adopts a broad view of stance as “personal feelings, attitudes, value judgements, or assessments”, and any expression of the speaker’s stance is considered to be an act of stance-taking. Following DuBois’ Stance Triangle, stance-taking is a three-part act whereby speakers evaluate a stance object, position a subject in terms of that stance object, and align stances between interlocutors. As such, there are limitless ways in which stance can be expressed. Though M3D does not aim to create an extensive list of how stance can be encoded multimodally, we have indeed identified the main types of stance studied so far. For example, within the parts of evaluating and positioning, one can talk about Affective stance, which conveys the emotional polarity of the speech act; Epistemic stance which conveys aspects of certainty, uncertainty, approximation, and evidentiality; or Politeness stance which refers to showing politeness or impoliteness as well as stance softening. In terms of alignment, we have also identified aspects such as agreement, which includes direct agreement, disagreement, confirmation, incredulity, obviousness, and cooperation which includes checks for understanding and backchanneling. Here is an example of a gesture which marks epistemic stance and evidentiality. The speaker is talking about how Darwin describes coming up with the idea for Natural Selection. He says “He was reading Malthus, On Population, and all of a sudden the basic algorithm of natural selection kind of pops into his head, and he says “oh, at last, I have a theory with which to work”. After saying this, the speaker says  [see example]. As you can see, the gesture acts to reinforce the idea that this claim is true and there is evidence for it in Darwin’s autobiography.  

The fourth pragmatic domain refers to Discourse organization. This refers to any gesture that conveys information about discourse structure or information structure. Gestures marking discourse structure may include things like marking parenthetical digressions, anaphoric marking, marking the beginning or ends of discourse sequences, or marking items in a list or a sequence of events. For example, the second gesture from this clip shows a gesture that is visually illustrating her discourse structure [see example]. The speaker is telling the audience she will return to the main topic of her speech by the end. Here is another example where three gestures are functioning to highlight the list structure of the discourse [see example]. 

In terms of marking information structure, these may include gestures that function to mark new or contrastive elements in the discourse. For example, here we see two gestures that mark a contrast [see example]. The speaker is talking about how building a Rube Golberg machine has its limitations but is also pretty wide open. He highlights these two contrasting ideas through the use of his referential gesture production, by pictorially illustrating a narrow distance with his hands to represent limitation and wider hands for something being open.   

The fifth and final pragmatic domain is interactional marking. This refers to gestures that are used to regulate discourse interactions between two interlocutors, such as the negotiation of turn-taking. For example when speaking, I may hold my hand up to stop the listener from interrupting me, or alternatively a listener may do a gesture to indicate their desire to take a turn.

It is important to highlight that we do not see these categories as being mutually exclusive, and indeed there may be quite a bit of overlap across these sorts of categories. For example, that turn-taking gesture that I did earlier could also be seen as a stance-marking gesture (aligning the stances between speaker who has the turn, and listener who wishes to take a turn) as well as a speech act (a request to take a turn). Not only may there be a bit of overlap, but more specifically a single gesture may convey multiple pragmatic meanings. For example, here is a gesture that marks a speech act, marks stance, and marks information structure [see example]. First, the speaker’s gesture reinforces the directive that the addressee bothers someone else. The form of the gesture is also a typical “shoo” or “go away” gesture which thus conveys the speaker’s affective stance, in that she is annoyed. Finally, the gesture is produced on Leonard Cohen, a discourse referent that contrasts with the speaker, in that she is saying “bother him, not me”.     

How to annotate the pragmatic domain

The pragmatic domain is annotated with the audio turned on, as it is necessary to have the context of speech to determine how the gesture is pragmatically functioning. Much like for the annotation of gesture referentiality, the pragmatic annotations should align with the stroke of the gesture. That is, for every gesture stroke, we should assess pragmatic meaning, and annotate over the interval of that stroke. In the M3D ELAN template, you will see a parent tier called pragmatic domain one - underneath there are a series of child tiers where you may optionally annotate more specific functions. For example, a gesture that is marking a contrastive focus, you may put “Discourse Organization” in the pragmatic domain tier, and “Contrastive Focus” in the pragmatic function tier. When a gesture conveys multiple pragmatic meanings, the Pragmatic Domain tier can be duplicated and renamed, so that there is a Pragmatic domain two, pragmatic domain three, etc. In this way labelers can annotate as many pragmatic functions as necessary.  

In this video, we’ve explored another aspect of the meaning dimension of gesture, specifically the different pragmatic domains. Remember: any gesture regardless of their referentiality may or may not convey a discourse-pragmatic meaning. We have identified 5 main pragmatic domains (Speech act marking, Operational marking, Stance marking, Discourse organization marking, and interactional marking), as well as a number of functions within each domain. A single gesture may convey multiple pragmatic meanings, which can be labeled with the M3D template. Thanks for watching!