Challenge: Identifying Gesture Form

Welcome back. In this video, we will cover the form dimension of gestures. We’ll start with why we’re looking at the form as a separate dimension, talk about the specific form subdimensions of handedness, hand shape, palm orientation, trajectory shape, and trajectory direction, and look at a few examples together. There are other form subdimensions in the M3D system, but we won’t cover them in this video.

In previous videos, we mentioned that the gesture unit, or G-Unit, covers the span of time from when the hands leave a rest position to prepare to carry out a gesture, through the end of the recovery to a rest position. Now, we want to explore what is kinematically happening within these gesture units. 

The form dimension of hand gestures refers to the arrangement of the hands and their movement patterns, sometimes called the articulator configurations and the kinematics. It is important to note that the movements of the arms, face and other body parts can also make significant contributions to a communicative act. In this video, we will only focus on the hands. 

Handedness

Let’s start with the subdimension of handedness. Handedness, which is coded on the “Manual articulator” tiers, refers to which hand or hands are being used in the gestural movement. It can be the left hand, which we label with the abbreviation “LH” or the right hand, labeled with “RH”. When both hands are used, we use “BH”. 

Manual articulator configuration

Next, let’s look at the manual articulators, the subdimensions of hand shape and palm orientation. Now before we start  looking at what hand shapes are, we need to consider palm orientation., Because, when we separate out the direction the palm is facing, from the shape of the hand we can greatly reduce the number of different hand shapes a labeler would need to recognize. 

Here are some examples of hand shapes. Let’s make a point and start with the deictic handshape which, in the labeling scheme, we would annotate with the letter “D” for “deictic”. When we bring that finger down and clench, we get a fist handshape , labeled as F for “fist” ; gently loosening it up we get a relaxed handshape which we label with R for “relaxed”. From here, straighten out those fingers as if you were getting ready to clap, and you have the open hand shape, labeled with the letter  O for “open”. Keep the fingers straight and bring them together, and you will have the knife handshape, labeled with K, for “knife”. Curl your fingers so all the tips touch, you get a hole handshape for a monocular, labeled with H for “hole”. Don’t worry about memorizing all of these right now. You can find a description of each label directly in the M3D ELAN template, in the controlled vocabulary. You can also check out the MIT Speech Communication Group’s Gesture Coding Manual for more examples

As I mentioned earlier, we separate the palm orientation from the handshape subdimension. You may notice that the perceived intention of a hand shape can look verydifferent when you change the palm orientation. Let’s take the knife handshape. With the palm orientation facing “forward”, you may noticed a different intention from the palm facing toward the speaker – labeled as “self” . Now here’s palm “up”, “down”, “left”/“right” toward the other side of the body, and then there’s “head”. When assessing palm orientation, it’s helpful to imagine an arrow pointing straight out of the speaker’s palm. So palm orientation is simply the direction in which that arrow is pointing.

Kinematics, or movement pattern

Now, let’s consider the movement path by looking at the subdimension of trajectory shape. Trajectory shape is labeled for the shape of the movement during the stroke. This can be straight, often up-and-down, curved , or looping which is a set of at least 2 closed and fluid oval shapes. 

Finally, we can talk about trajectory direction. This simply refers to the directions in which the gesture is moving. These refer to directions you are already familiar with and we label them as such. So for trajectory direction, we have up, down, to the speaker’s left or right, forward, or towards the speaker’s self. We also have compound directions, like when combining forward and down,  or any other combinations.

You may notice that the hand can sometimes only appear to be moving because it is connected to a larger moving body part. For example, when I rotate my torso my hands seem to move, but it’s only my torso that’s moving. So we do not annotate these cases as hand movements.

Now, in practice, we recommend annotating the form tiers over the duration of the G-Unit without audio. This will provide more thorough annotations, which then helps guide you for annotating the other aspects such as gesture phasing. However, you may choose a different convention based on your research questions, such as annotating form over only the duration of the gesture stroke, or by only annotating particular tiers within the form dimension. As you choose your conventions, it is useful to clearly communicate them, because these form subdimensions can look different when including, or when not including the gesture phases

In this video, we’ve explored the form dimension, with its subdimensions of handedness, hand shape, palm orientation, trajectory shape, and trajectory direction, and how we annotate them in M3D. 

As a reminder, Annotating the Gesture Form takes place with the audio muted. Thank you for watching!