Gesture and Relation to Speech

The relationship between the gesture and the spoken context is the most important tool for determining the role of gesture in the communicative context. The coding of speech we use is based on taking each utterance as a separate unit and investigating parts of speech and syntax on the level of the utterance. For this reason, it is important that only the immediate spoken context of a gesture be used to determine the relationship between gesture and speech. Thus, if a child uses a gesture to answer a question asked by the mother, the role of the gesture must only be considered in relation to the child’s accompanying speech rather than the discourse as a whole. This should be kept in mind when coding the gesture-speech relationship, but also the thematic role of the gesture.

Gesture Types

Code	Gesture Type	Definition	Sample Forms
DP	DEICTIC-POINT	Indication of a distal object through directional extension of the finger, hand, limb, or occasionally, head.	point, palm, cont point
DS	DEICTIC-SHOW	Drawing attention to an object or body part by holding it up towards the interlocutor.	hold
C	CONVENTIONAL	Culturally-determined or ritualized gestures that serve as symbolic replacements for ideas and are produced in a similar form every time.	nod, shake, come, flip, dismiss, wait, shh, thumbs up, tada
G	GIVE	Requesting an object by reaching towards the interlocutor with the hand palm-up.	give
S	BABY SIGN OR ASL	Gesture is a sign from either baby sign or ASL.	Milk, more (BS) Alphabet (ASL)
R	REPRESENTATIONAL	Depicting attributes or actions associated with objects through spontaneous (mostly manual) movement.	iconic, metaphoric
E	EMPHATIC	Simple manual gestures or nods marking the prosody of speech through temporal correlation with particular words or syllables.	beat, nod
FA	FUNCTIONAL ACTS	Gesture-like movements that reenact desired actions (demonstrators) or start object transfers (hold out).	demo, hold (take)

Non-Literal Points (2008 Addition)

Deictics typically refer to the object that they point to,” but some deictics may be “non-literal” in that they use a location or object to refer to a different one. This may occur as a convention by using one object to stand for another (e.g. pointing towards dad’s study to mean dad though he is never in the room during the gesture), or as a spontaneous use of nearby objects to reference more distal or out of view objects (e.g. pointing towards own shirt to indicate the spot on another’s shirt where there is a stain). A point should be considered to be non-literal when it does not refer to either the specific object indicated, or to that class of objects.

Where non-literal points are used, the suffix ”.nl” is added after the Gesture Type, making the code DP.nl. The .nl suffix is only added to the gesture type column.

Examples

Utterance + GESTURE	Semantic Correlate:
“She got cut right here” + POINT to own knee	HERE
“Your father will be angry when he gets home” + POINT to Dad’s chair	FATHER

Gesture-Speech Relation (non-iconics)

Bolded cells = common categories

	Code	Gesture’s relation to speech	Definition	Examples
Reinforcing	RF	Reinforcing	Gesture reinforces the info provided in speech.	“Cat” +[point to cat]
	RF.a	Reinforcing attribute	Gesture reinforces an adjective in speech describing perceptual features.	“Here is a blue one” +[point to blue block]
	RF.p	Reinforcing predicate	Gesture reinforces the spoken predicate describing an action the referenced object performs.	“Crying” +[point to crying baby in book]
	DA	Disambiguating	Gesture disambiguates speech	“Put it there” +[point to table] “These are mine” +[palm toys]
Adding	ADD	Adding information	Gesture adds info to speech such that G+S = one proposition.	“Cookie” +[give gesture] “Look” + [point to book]
	ADD.s	Adding info to a related sound	Speech adds related meaningful sound.	“Meow” +[point to cat]
	ADD.ns	Adding info to unrelated sound	Speech adds unrelated meaningful sound.	“Uh-oh” +[point to cat] “Mm-hmm” +[point to table]
	ADD.nr	Adding info to unrelated utterance	Gesture adds a new proposition or speech and gesture conveys unrelated information such that G+S = two propositions. Used only with give, come, wait, and other gestures that have the meaning of a predicate.	“Show me your toy” + give
	ADD.err	Matching error	Gesture and speech refer to same object but one incorrectly labels it (occurs mainly in infants)	“Cat” +[point at dog]
	ADD.err.s	Matching error at the sound level	Gesture and non-word sound refer to same object but one incorrectly labels it (occurs in infants)	Meow” +[point at dog]
	ADD.f	Adding info to a filler expression	Gesture accompanies a “filler expression” (conventional utterance) that provides little literal information about the exchange. Fillers typically accompany the “give” gesture.	“OK/yeah/thank you/here you go” + [give gesture]
	ADD.q	Adds info answering a spoken question	Gesture provides information (such as a location or yes/no info) that answers the question posed in speech.	“Where is the bird?” +[point to tree] “Is this the one?” +[head shake]
Emphasizing	E.b	Emphasizes with a beat	Hand, arm, or head movements where the gesture stroke synchronizes with emphasis in the speech.	“What I’m SAYING is +[hand beat on “saying”]
	E	Emphasizes without beat	Gestures that act like discourse markers, by emphasizing what is said.	“Hard to decide” +[head shake] “All of them” +[nod]
	MS	Adds to a meaningless sound	Gesture is accompanied by a meaningless sound (occurs almost exclusively in infants).	“Da!” +[hold up toy truck]
	UC	Unclear relationship	The gloss for the gesture does not provide enough information or the speech is indecipherable.	“I want ####” +[point to toy]
	X	No speech	Gesture is produced on its own.	[point to desired object]

How to choose G-S relationship

When an object label is used in speech, the gesture-speech relationship will most often be reinforcing (RF). Gestures can disambiguate speech when there is a deictic word such as this, that, here, there, it, or a personal pronoun (e.g., he, she, them, etc.) An ADD code will be used when there is no object label and no deictic in speech.

Superordinate Categories

Superordinate categories can be reinforced. For example, animal, toy, clothes, and food are all categories that can be reinforced in speech by gesturing towards any item in that category. Property categories such as color, shape, and size cannot be reinforced, but specific items within these categories (e.g., red, round, or big) can be reinforcing an attribute if the object label is not present.

Speech	Gesture	GS_relationship	Semantic Correlate
Give me the animals.	Point to grizzly bear	RF	animals
My room is messy because my clothes are all over	Palm clothes	RF	Clothes
What shape?	Continuous point to stamps of various shapes	ADD	X
What color do you want?	Hold markers of various colors	ADD	X
What color is this?	Hold blue marker	DA	This
What color?	Hold pink marker	ADD	X
Want pink?	Hold pink marker	RF.a	pink