Resources 

Useful Websites


ELAN 

https://www.ling.upenn.edu/~wlabov/L560/ELAN_introduction.pdf


Praat 


Video/audio editing software


Transcription/alignment tools


Motion Tracking / Post-annotation analyses

Collection of the mentioned resources (organized by challenge)

Learning about M3D

M3D Resources (Labeling Manual, ELAN Template, and the M3D-TED Corpora) - https://osf.io/ankdx/

Kendon, A. (2004). Gesture: Visible action as utterance. Cambridge University Press.

McNeill, D. (1992). Hand and mind: What gestures reveal about thought. University of Chicago press.

McNeill, D. (2005). Gesture and thought. University of Chicago Press.

Getting started with ELAN 

Rosenfelder, I. (2011). A Short Introduction to Transcribing with ELAN. 

https://www.ling.upenn.edu/~wlabov/L560/ELAN_introduction.pdf


ELANport (a series of tutorial videos on using ELAN): https://beyzasumer.com/elanport/

ELAN website


A note about the assessment of annotations 

Artstein, R., & Poesio, M. (2008). Inter-coder agreement for computational linguistics. Computational linguistics, 34(4), 555-596.

Gullberg, M. (1998). Gesture as a communication strategy in second language discourse: A study of learners of French and Swedish. Lund University Press.

Passonneau, R. (2006). Measuring Agreement on Set-Valued Items (MASI) for Semantic and Pragmatic Annotation. Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC ’06). http://www.lrec-conf.org/proceedings/lrec2006/pdf/636_pdf.pdf 

Identifying Gesture Form 

MIT Speech Communication Group’s Gesture Coding Manual - https://speechcommunicationgroup.mit.edu/gesture/coding-manual.html

Identifying Rhythmic Properties:  Beat-like-ness

McNeill, D. (1992). Hand and mind: What gestures reveal about thought. University of Chicago press.

McNeill, D. (2005). Gesture and thought. University of Chicago Press.

Tuite, K. (1993). The production of gesture. Semiotica, 93(1–2), 83-105. https://doi.org/10.1515/semi.1993.93.1-2.83 

Identifying the Pragmatic Domain
Du Bois, J. W. (2007). The stance triangle. In R. Englebretson (Ed.), Stancetaking in discourse: Subjectivity, evaluation, interaction (Vol. 164, pp. 139–182). John Benjamins Publishing Company. https://doi.org/10.1075/pbns.164.07du

Kendon, A. (2017). Pragmatic functions of gestures: Some observations on the history of their study and their nature. Gesture, 16(2), 157–175. https://doi.org/10.1075/gest.16.2.01ken

Publications/Corpora 


Publication about M3D:

Rohrer, P. L. (2022). A temporal and pragmatic analysis of gesture-speech association: A corpus-based approach using the novel MultiModal MultiDimensional (M3D) labeling system [Doctoral dissertation]. Universitat Pompeu Fabra & Nantes Université. https://theses.hal.science/tel-03994053/


M3D-TED corpora:


Gilbert, E. (2009, February). Your elusive creative genius. [video]. TED Conferences. https://www.ted.com/talks/elizabeth_gilbert_your_elusive_creative_genius


Johnson, S. (2010, July). Where good ideas come from. [video]. TED Conferences. https://www.ted.com/talks/steven_johnson_where_good_ideas_come_from


Keith, D. (2007, September). A critical look at geoengineering against climate change. [video]. TED Conferences. https://www.ted.com/talks/david_keith_s_surprising_ideas_on_climate_change


Sadowski, A. (2010, April). How to engineer a viral music video. [video]. TED Conferences. https://www.ted.com/talks/adam_sadowsky_how_to_engineer_a_viral_music_video 


Stewart, M. G. (2010, February). How YouTube thinks about copyright. [video]. TED Conferences. https://www.ted.com/talks/margaret_gould_stewart_how_youtube_thinks_about_copyright


Stone, E. (2016, October). Academic research is publicly funded - why isn't it publicly available? [video]. TED Conferences. https://www.ted.com/talks/erica_stone_academic_research_is_publicly_funded_why_isn_t_it_publicly_available 


TEDx Talks. (2014, June 12). Ca donnerait quoi si on prenait des cours de cerveau ? | Kevin Finel [Video]. YouTube. https://www.youtube.com/watch?v=ksB80XYiYzI


TEDx Talks (2015, April 14). Comment oser prendre des risques pour vivre une vie intense ? | Frederique Bedos [video]. YouTube. https://www.youtube.com/watch?v=zjEfmB0DV0k


TEDx Talks (2016, July 5). Et si votre rêve devenait possible... | David Laroche [video]. YouTube. https://www.youtube.com/watch?v=fbGfe78_2jc


TEDx Talks. (2018, March 24). L'amour dure 90 secondes | Marine Duvouldy  [Video]. YouTube. https://www.youtube.com/watch?v=9fxo9YJhnG8


TEDx Talks. (2018, December 18). Négociation: ne cherchez pas le compromis | Julien Pelabere [Video]. YouTube. https://www.youtube.com/watch?v=N9duDfWSfU4




M3D applied:

Florit-Pons, J., Vilà-Giménez, I., Rohrer, P. L., & Prieto, P. (2023). Multimodal Development in Children’s Narrative Speech: Evidence for Tight Gesture–Speech Temporal Alignment Patterns as Early as 5 Years Old. Journal of Speech, Language, and Hearing Research, 66(3), 888–900. https://doi.org/10.1044/2022_JSLHR-22-00451 

Meissl, K., Sambre, P., & Feyaerts, K. (2023). Contrast Pairs in Orchestra Instruction as a Window into the Multimodal Expression of Stance. In Papers of the Linguistic Society of Belgium (Vol. 16, pp. 1–16). Belgische Kring voor Linguïstiek. 

Rohrer, P. L., Florit-Pons, J., Vilà-Giménez, I. and Prieto, P. (2022). Children Use Non-referential Gestures in Narrative Speech to Mark Discourse Elements Which Update Common Ground. Frontiers in Psychology. 12:661339. http://doi.org/10.3389/fpsyg.2021.661339 

Rohrer, P. L., Prieto, P., & Delais-Roussarie, E. (2022). Le rythme prosodique guide le rythme gestuel. XXXIVe Journées d’Études sur la Parole - JEP 2022, 588–596. https://doi.org/10.21437/JEP.2022-62 

Vilà-Giménez, I., Florit-Pons, J., Rohrer, P. L., Coego, S., Gurrado, G., & Prieto, P. (2023). Audiovisual corpus of Catalan children’s narrative discourse development. https://doi.org/10.17605/OSF.IO/NPZ3W 

Vilà-Giménez, I., Demir-Lira, Ö. E., & Prieto, P.  (2020). The role of referential iconic and non-referential gestures in children's narrative production: Iconics signal oncoming changes in speech. Proceedings of the 7th Gesture and Speech in Interaction (GESPIN). KTH Speech, Music & Hearing and Språkbanken Tal. Stockholm, Sweden.


M3D cited (selected works):

Fisher, V. J. (2021). Embodied Songs: Insights Into the Nature of Cross-Modal Meaning-Making Within Sign Language Informed, Embodied Interpretations of Vocal Music. Frontiers in Psychology. 12:624689. http://doi.org/10.3389/fpsyg.2021.624689 

Kosmala, L. (2021). A multimodal contrastive study of (dis)fluency across languages and settings: Towards a multidimensional scale of inter-(dis)fluency [Doctoral dissertation]. Université Sorbonne Nouvelle - Paris 3. http://doi.org/10.13140/RG.2.2.28161.58726 




Please be sure to cite M3D when applying it to your own data (see below). If you have used or described M3D and have cited us, please let us know by contacting us so that we can include your publication on our website.