Joyrex Labs

Outline

This Page

Representational Gesture Codes

Representational Gesture Type

Code Gesture Type Definition Examples
R.a Representing-attribute Iconic gesture characterizing an entity by means of specifying an attribute (size, shape) or a part associated with the entity (rabbit’s ears, lion’s claw), or the entity itself (frog, duck). BIG (open arms wide to indicate size of fish) RABBIT (hold fingers up as if they are the ears of a rabbit) AIRPLANE (lift arms up sideways and make a motor sound as if an airplane) EVERYWHERE (sweeps around in circle) ROUND (draw a circle with finger in air)
R.m Representing-motion Iconic gesture characterizing an entity by means of an action associated with the entity. EATING (move hand back and forth into mouth as if eating) JUMPING (jump up and down to mimic a frog’s jumping motion) STINKY (fanning hand back and forth in front of face) +traceshape
R.d Representing-direction Iconic gesture characterizing the direction of the motion UP (move finger upwards to indicate upwards path) +tracepath AROUND (move hand in a circle to indicate around) +tracepath
R.met Representing- metaphor Representational gesture indicating a metaphorical object or movement “We’ll see how it goes” + drag hand slowly to side indicating passage of time “So, we can have this one or that one” + hands in catchment handshape in 2 different locations “It’s hard to decide” + hands indicating scales weighing options “Quiet” + hand presses down slowly

Metaphorics

Gestures that look like iconic gestures but do not literally represent an object or action are coded as “metaphoric” gestures. See examples on here and here. The imagery may be obvious in speech (e.g. “I have an idea” + hand cupped upward) or not obvious (e.g. “It’s a difficult problem” + fingers pointing up pinched together as if holding something small or fine). Metaphorics should be indicated in the gesture type column by R.met and again in the gesture-speech relationship column by RF.met or ADD.met. Unclear relationships are coded as UC and those occurring without speech are coded as X. All other columns should be coded the same as for iconics.

Iconic Gestures Metaphoric Gestures
“We need a basket” + hand cups upward in air “So I want to say...” + hand cups upward in air as if holding an idea
“The tall one” + hand raised above head palm down “We were arguing” + hands face one another and rotate back and forth in opposition
“He had a great hat” + hand placed on head “Think about it this way” + hands held apart facing one another rotate to switch positions as if moving an object to another angle
“Rolled down the hill” + hand makes small circles while moving out from body and down “After a while” + hand makes small circles while moving outwards from body as if showing progression
“She has a lot of lilies” + hand moves forward with fingers together and opens out like a flower “Then I understood” + hand moves forward with fingers together and opens out like a flower
“He hopped into the bush” + hand with index finger extended moves in arc from one point in air up out and down to another “This one or that one” + R hand presented palm up on one side of body and then repeated with L hand
  • The iconic and metaphoric gestures do not necessarily correspond within a row, but are provided to demonstrate differences between these two types of gestures.

Pretend Play and Emblems

Gesture-Speech Relationship during pretend play

Often during play sessions, a child or parent will use an object (a stuffed animal, a plastic cup) to stand for something else (a policeman, a rocket ship). In these cases, the gesture-speech relationship can become confusing when someone gestures to an object and give a contradictory verbal label (e.g. hand me the rocket + point to cup). Once an object has been established as standing for another object during a play session, then a gesture towards or about the object should be considered to be reinforcing (RF) the verbal label of either the real object or pretend identity. If a gesture occurs during the initial utterance establishing a pretend identity for an object, then the gesture is considered to be disambiguating if a pronoun is used (e.g. this can be the policeman + point to Elmo doll), reinforcing the true identity if the whole label is used (e.g. Elmo can be the policeman + point to Elmo doll), and adding the pretend identity if only the new label is given (e.g. policeman + hold Elmo doll).

Examples:

Utterance G-S relationship Semantic Correlate
“This one is the fire truck” + POINT (to box - first time) DA THIS ONE
“There’s the fire truck” + POINT (to box, second time) RF FIRE TRUCK
“It’s a fire truck” + POINT (to box, second time) RF FIRE TRUCK
“The box is a fire truck” + POINT (to box, second time) RF BOX
“Hand me the fire truck” + POINT (to box, second time) RF FIRE TRUCK
“Hand me the box” + POINT (to box used as fire truck) RF BOX
“This is a policeman” + POINT (first time, to Barbie doll) DA THIS
“A policeman” + POINT (first time, to Barbie doll) ADD X

Emblems [.e]

Children sometimes use gestures that seem to be conventional but are not shared with most other children in the sample. These idiosyncratic conventions will have the same form every time they are used and may take the place of a noun phrase, verb phrase, or expression rather than a single word. If an individual frequently uses exactly the same gesture to represent something, it should be coded as an emblem. These types of gestures are coded as iconics, but feel less spontaneous and more conventional than true iconic gestures. They may resemble ASL signs in their persistence of form, but are not considered conventional gestures.

  • e.g. Child holds up two fingers to represent a rabbit (R.a.e)
  • e.g. Child clenches fingers together to indicate a favorite blanket (R.a.e)

Representational Gesture-Speech Relationship

Code Gesture Type Definition Examples
RF.a Reinforcing-attribute Reinforces an attribute, body part, or entity conveyed in speech
  • ‘this big’ + BIG (open arms wide to indicate big size)
  • ‘it’s round’ + ROUND (draw a circle in air with finger)
  • ‘he’s a lion’ + LION (extend fingers to mimic a lion’s claw)
  • ‘I want balloon’ + BALLOON (hold curved palm as if it is a balloon)
  • ‘hat’ + HEAD (places hand on head)
RF.p Reinforcing-predicate Reinforces the action conveyed in speech
  • ‘wash’ + WASHING (make washing hand motions above head)
  • ‘bounce’ + BOUNCING (move hand up and down)
RF.d Reinforcing-direction Reinforces the directionality in speech
  • ‘up’ + UP (move index finger upward)
  • ‘around’ + AROUND (draw a circular path with finger)
RF.met Reinforcing-metaphor Reinforces the metaphor in speech
  • ‘this one or that one’ + SEPARATE ENTITIES (R hand presented palm up on one side of the body and then repeated with L hand)
  • ‘We were arguing’ + ARGUING (hands face one another and rotate back and forth in opposition)
DA.d Disambiguate-direction Disambiguates directionality
  • ‘go this way’ + TOWARD KITCHEN (trace path to kitchen)
  • ‘turn it the other way’ + CLOCKWISE (turning hand clockwise to open jar)
  • ‘walk this way’ + AROUND TABLE (trace path around coffee table)
ADD.a Add argument Adds another argument to speech (and in some occasions adds an attribute)
  • ‘Draw’ + CIRCLE (draw an imaginary circle in air)
  • ‘Roar’ + LION (extend arms with outstretched fingers to mimic a lion with claws)
  • ‘This much’ + LITTLE (pinch fingers to indicate small size)
  • ‘All the way to the sky’ + HEIGHT (horizontal palm indicating height of tower)
ADD.p Add predicate Adds another predicate to speech
  • ‘I like it’ + EATING (move hand back and forth into mouth to convey eating)
  • ‘He go’ + SLEEPING (rest head on upward facing palm)
  • ‘Bzz’ + FLYING (flapping arms like wings)
  • ‘I want to eat you’+ BITING (hand pinching M like insect would)
ADD.d Add direction Adds a path to speech
  • ‘Go’ + UP (trace an upward path with finger)
ADD.met Add metaphor Adds a metaphor to speech
  • ‘So I want to say’ + IDEA (hand cups upward in air as if holding an idea)
UC Unclear Gesture-speech relationship unclear  
X No speech Gesture on its own  

Reinforced or Added Characteristics

Goal: To specify what type of information is being either reinforced or added by the gesture.

Actions:

  • (M)=manner
  • (P)=path
  • (L)=location

Objects:

  • (SI)=size
  • (SH)=shape
  • (AB)=action by
  • (AO)=action on

Explanation

This measure attempts to capture which feature of an object or action is reinforced or added to the speech by a representational gesture. Gestures that reinforce speech should have a reinforcing value, but not an adding value. ADDs should be the reverse. Most iconic gestures will provide some information that is not present in speech, but if the main effect of the gesture is reinforcing, then it should be described as having a particular reinforcing characteristic.

Gestures that reinforce and add information

Occasionally a gesture will appear to reinforce something in speech but also add an argument or a predicate to speech through the movement of the gesture. In these cases, the gesture should have a code for both information reinforced and added.

When to code

Only code reinforced or added characteristics for representational gestures (iconics and metaphorics).

  Code Characteristic Definition Description
Actions M Manner The type of action performed or the way in which it is executed. Hand is bounced up and down to indicate jumping.
P Path The path through which an act moves as it is performed. Hand sweeps out and to the side to signify the path something has taken around another object.
  L Location The location (or relative location) of an act or object. Flat hands held palms down side by side to indicate that one object is close to another.
Objects SI Size The size of an object. Hand extends over head to signify “tall”
SH Shape The shape of an object. Finger traces “v” in air to signify the type of shirt desired.
AB Action by An object referenced by an action it typically performs. Hands form claws, raised up and swept out and down to signify “bear”
AO Action on An object referenced by an action typically performed on it. | Flat hands with palms together, hands open out to signify “book”

Examples

Utterance Gesture Reinforced Added
“I followed him” Hand traces curving path horizontally out from body   P
“I ran after him” Arms pump as though running. M  
“I found it” Hands form cube shape in air.   SH
“I found the earring” Index finger and thumb narrow towards one another. SI  
“Then it floated up” Hands curved and held near one another as if around a ball or balloon and rise upwards. P SH
“Then it floated” Hands curved and held near one another as if around a ball or balloon and rise upwards.   P,SH
No Speech   X X

Perspective (1st vs. 3rd person)

Goal: To determine when iconic gestures are produced from a character (1st person) versus and observer (3rd person) viewpoint.

Explanation

The distinction between first person and third person gestures is concerned with what viewpoint the gesturer takes when creating a gesture. When the gesturer is using his hands as hands during a gesture (e.g. cupping them around an imaginary object), or using gesturing about directionality in relation to their own axis (e.g. moving a hand over one’s shoulder to indicate “behind”) then the gesture is considered a first person gesture. When, on the other hand, the gesturer uses his hands to represent an object (e.g. fingers scissoring to represent “person running”) or a relationship between objects (e.g. one hand on top of the other to represent “on top”), the gesture is considered a third person gesture.

Code Perspective Definition Examples
1 First-person Gesture is produced as though the gesturer is performing the action or embodying the object. Hands expand out from waist to signify “fat” Hands in fists trace path of opening a book
3 Third-person Gesture is performed as though the gesturer is looking at the action or object from a distance. Hands trace round shape in gesture space to signify fat Hands flat together fall open like the pages of a book
G Global Concept is one that cannot have a 1st-person perspective. Hands sweep out and around to signify “everyone”
UC Unclear Perspective of gesture is unclear (could be 1st or 3rd person) Hand reaches up in air to signify “above” but is neither clearly over gesturer’s head nor in relation to other object
X Not applicable Gesture is non-iconic Deictics, Conventional, Functional Acts.
  • Perspective should be coded regardless of whether or not speech accompanies the gesture.