Point OutWordsTM: iPad software to develop motor communication skills in people with severe autism
Update: Apple unilaterally have taken Point OutWords off the App Store until such time as we can update its look and feel to conform to their latest standards, which are a moving target. At the current time we have no funds to do this. Point OutWords is an open-source project and we would gratefully accept voluntary work from an experienced iOS programmer (contact: belmonte at mit dot edu).
Point OutWordsTM is distributed under the terms of the Mozilla Public License 2.0; it’s free and always will be. The source code is available here; you can get it with git clone https://bitbucket.org/belmonte/point-outwords.git . If you wish to refer to Point OutWordsTM, you may cite the following article:
Coming soon (this summer) check back here for a preprint of our paper on the results of the NIHR feasibility trial, which expands on the APS 2021 presentation below.
Point OutWordsTMhas not yet been proven effective in developing communicative skills and might not have any benefit at all, although we hope that it will.
Everybody talks about autism, everybody knows someone living with autism, but the science and the public rhetoric of autism are dominated by that fraction of autistic people who can speak to communicate. Those whose autism is more severe are all too often overlooked: they are excluded from scientific research as unable to comply with experimenters' instructions, and they get the short end of the stick when it comes to community placements and education. They are the most in need of clinical therapeutic and educational interventions, yet all too often they get left out of the picture.
Our and others' research has indicated that a substantial fraction of people with autism who lack communicative speech - fully one third of them, in our clinical sample - also experience severe deficits in the cognitive planning and motor execution of body movements. In this group, receptive language skills outpace expressive language skills: people with autism who have learnt to type report that they've spent years listening in as the world turns around them, and learning from it, but have been tragically and frustratingly unable to respond because they cannot make their mouths and vocal tracts move to translate thoughts into sequences of sounds.
Point OutWordsTM, designed in collaboration with autistic clients and their communication therapists at the Com DEALL Trust, exploits the autistic fascination with parts and details to motivate attention to learning manual motor and oral motor skills essential for communication. Along the way, autistic clients practise pointing and dragging at objects, then pointing at sequences of letters on a keyboard, and even speaking the syllables represented by these letters. Point OutWordsTM is free software, and always will be.
Along with its communicative deficits, the autistic cognitive profile also includes many unusual strengths. One of these is attention to localised details. Many people with autism excel at assembling jigsaws and similar spatial puzzles because they are so skilled at perceiving the details of the matching edges on adjacent pieces. Whereas many teaching and learning strategies adapted from methods for non-autistic people end up working against autistic cognition by asking people with autism to do what they cannot easily do, Point OutWordsTM works with autistic cognition, by beginning from this point of strength. Point OutWordsTM is intended to be used with a therapist, teacher or caregiver who can support the autistic client in an 'errorless learning' format, filling in responses and correcting errors when the client is unable to do so independently, and redirecting the client towards the task when attention wanders.
Point OutWordsTM contains three modes: Point, Type, and Speak. Learning begins in Point mode, which teaches the client how to point and drag within a touchscreen interface: The client selects an object from one of several scenes depicting activities of daily living, for instance a shampoo bottle, a cup, a birthday cake, a toilet. The object is segmented into jigsaw puzzle pieces which are scattered round the touchscreen. Using pointing and dragging movements, the user must assemble these pieces into a complete whole. The touchscreen's software interface is adapted so as to accommodate an autistic style of motor control, in which difficulties in eye-hand coordination may make points of initial contact with the touchscreen only approximate, and dragging movements might contain discontinuities during motor errors when the finger breaks contact with the touchscreen. This activity not only trains manual motor skills, but also teaches the cognitive notion that an object can be associated with a sequence of things - in this case, a sequence of puzzle pieces that fit together to make up the object.
Type mode takes advantage of this notion of object as sequence to develop an association between the object and the sequence of letters in the word for that object. In Type mode, puzzle pieces cannot be dragged into place in an iconic style of reference, but instead must be cued by symbolic reference: each puzzle piece is labelled with a letter in the word, there are as many puzzles pieces are there are letters in the word, and in order to cause a piece to snap into place within the puzzle, the client must press the corresponding letter key on a virtual keyboard that appears on the touchscreen. Although most clients initially respond to this challenge by perseverating in attempting to drag the pieces iconically into place as they did in Point mode, after some experience with Type mode in an errorless-learning format they learn to press the corresponding key. When they make this transition from pointing at the object itself to pointing at the keyboard, they've begun to master symbolic communication - and all of this has been motivated from their cognitive strength at detail-oriented perception!
For clients who are able to vocalise, Speak mode offers an opportunity to practise the motor skills that support spoken communication. In this mode, the object is segmented into a number of puzzle pieces equal to the number of syllables in the word. The client is prompted to speak each syllable, and OpenEARS speech recognition software is used to recognise this pronunciation, with a high tolerance for errors of speech articulation. Speaking a syllable causes the corresponding puzzle piece to snap into place. Thus, in all three modes, Point OutWordsTM provides a motivating, game-like framework within which clients can practise these manual motor and oral motor skills that are prerequisite to communication.
Videos of Point mode and Type mode. Point OutWordsTM is applied in an errorless-learning format in which the therapist or caregiver models and scaffolds appropriate responses, then fades this assistance as clients begin to initiate appropriate responses on their own.
Point OutWordsTM logs all prompts and stimuli, all contacts with the touchscreen, and even all movements of the iPad (using the iPad's built-in accelerometers). These logs can be used not only for monitoring therapeutic progress but also for behavioural scientific research. For example, one method of typed communication previously developed for autistic clients has been shown to be untrustworthy because it is liable to unconscious influence by a therapist who subtly moves the keyboard. Despite this fallibility, individual case reports exist of non-speaking people with autism who've learnt to type independently of any therapist and whose communications therefore appear to be genuine. It's thus crucial to know, then, both in general and for specific client-therapist pairs, whether, to what extent, and under what circumstances typed communications from people with autism are genuine and reliable. When typing takes place through Point OutWordsTM, the accelerometer logging shows whether - and when, and in which directions - the iPad is moving. Clients can opt in to allow their anonymised logs to be uploaded to a central server for use in research.
Two screenshots from a development version of Type mode. In the first frame, the client has perseverated in pointing iconically at the puzzle piece, and is being prompted to point at the key. In the second frame, the client successfully points to the corresponding key.
This report is independent research funded by the National Institute for Health Research Research for Patient Benefit, “Evaluation of Point OutWords, a Motor Skills Intervention to Promote Language Development in Non-Verbal Children with Autism: A Feasibility Study”, PB-PG-0816-20019. The views expressed in this publication are those of the author(s) and not necessarily those of the NHS, the National Institute for Health Research or the Department of H
ealth.
