Assessing autism the ADTree classifier way

The Autism Diagnostic Observation Schedule (ADOS) has a hallowed place in autism research and practice. At 23 years of age, ADOS and its counterpart, the Autism Diagnostic Interview (ADI), have pretty much cornered the gold-standard autism assessment market over the years. ADOS in particular with its sliding scale of modules covering the spectrum of language and developmental ability and generous assortment of 'props', resides in quite a few cupboards of child development centres and research institutes worldwide.

One of the nice things about ADOS aside from the hands-on, interactive tasks particularly in the more 'early-years' modules, is the way that it valiantly attempts to standardise behaviour and codes autism, autism spectrum or not via the use of an algorithm. This paper headed by one of the primary architects of ADOS, Cathy Lord, summarises the development of that algorithm based on the ADOS-G (generic). Said algorithm having fairly recently been revised by Katherine Gotham and colleagues to amalgamate some features (communication + social interaction = social affect) potentially tied into the proposed revisions for autism diagnosis in DSM-V.

Perfect you might say, so if it ain't broken why fix it? Well, gold standard or not, ADOS can be quite time-consuming to deliver (but not as time-consuming as the ADI I might add). Assessors also need to be properly trained and kept up-to-date with their ADOS training, and despite all its standardised prowess, people are people and sometimes mistakes are made - which is partly why ADOS is an assessment tool and not an all-encompassing diagnostic tool. Of course there are other 'issues' that have been raised with ADOS in mind (e.g. comorbid ADHD interfering, assessing Asperger syndrome) but I'm not here to start that critique.

Enter then a study by Dennis Wall and colleagues* (full-text) and their suggestion that based on some clever machine-learning algorithms, 8 out of the 29 items normally used in the delivery of a module 1 ADOS (no speech present) might just have the ability 'to classify autism' with complete or near 100% accuracy.

• Wall and colleagues constructed 16 possible machine-learning algorithms based on the 29 coding items of the ADOS module 1. For anyone really interested, these 29 items are spread across several domains including: (a) language and communication (9 items including vocalisations, response to name, pointing and use of gestures), (b) reciprocal social interaction (11 items including eye contact, giving, requesting, showing and response to joint attention), (c) play (2 items - functional object play and imagination), (d) restricted, repetitive behaviours and interests (4 items including unusual sensory interests and hand / finger mannerisms) and (e) 'other abnormal behaviours' (3 items including overactivity and anxiety). I should point out that when it comes to the final algorithm, only 17 of these 29 items are actually used (at least according to the original pre-Gotham algorithm - the revised algorithm uses 14 items I think??) and even then, only 12 items contribute to the autism / autism spectrum cut-off scores used based on items in the communication and social domains.
• Based on the use of data from a group of children with autism with module 1 ADOS scores from the AGRE dataset (autism: n=612; non spectrum: n=11), 90% of participants were used as part of a training set and the remaining 10% as a tester set as part of the proposed algorithms (much the same way that other studies have used). 
• Further validation of the best classifiers was undertaken based on other independent samples who reached cut-offs from module 1 ADOS to include data from the Boston Autism Consortium (autism: n=110; non spectrum: n=4) and the Simons Simplex Collection (autism: n=336; non spectrum: n=0).
• Although 2 out of the 16 algorithms "operated with perfect sensitivity, specificity and accuracy", one of them, the ADTree classifier, was selected as the best option because it relied on only 8 items of the module 1 ADOS to produce such results (the alternative used a whopping 9 items). Validation using the Autism Consortium and Simons Simplex Collection data correctly classifying all but 2 participants (from the Simons collection) who exhibited "... the potential presence of non spectrum behaviours".
• Drum-roll for the 8 distinguishing module 1 items isolated: (i) frequency of vocalisation directed to others, (ii) unusual eye contact, (iii) responsive social smile, (iv) shared enjoyment in interaction, (v) showing, (vi) spontaneous initiation of joint attention, (vii) functional play with objects and (viii) imagination/creativity. Use of these 8 items would also reduce the number of activities needed to elicit behaviours; so goodbye 'response to name' and 'response to joint attention'. I'm glad to say that the 'birthday party' activity remained!

Despite the obvious issues concerning a reliance on pre-ADOSed children and the use of only a handful of non spectrum controls, I find myself very interested in these results and the performance of the ADTree classifier. As per the study conclusions "The ADTree classifier consisted of eight questions, 72.4% less

than the complete ADOS Module 1". Purely from a practical point of view and the quite significant levels of concentration needed to manage tasks and elicit scores from the original module 1 ADOS, there is an obvious advantage to focusing on 8 items as opposed to 29 items. I do wonder exactly how much time will be shaved off an assessment bearing in mind that the majority of module 1 activities still need to be covered in order to elicit proper responses to items. So activities like 'free play', 'functional and symbolic imitation' and 'birthday party' are still required and can often take up the lion's share of assessment time.

Whilst the 8 selected items are interesting, there are some notable absences from the classifiers based on things like issues with pointing, the use of gestures and the use of facial expressions directed to others. I was always under the impression that these were important facets of autism, or at least autism in the early years but this study suggests not so in terms of classification. What exactly this might say for other results linking pointing, proto-declarative pointing for example, to autism is source for speculation. 

Homing in on those classifier items also opens a number of doors to things like quick and early screening and even where attention perhaps needs to be focused with regards to measuring outcome - that is how maturation and/or intervention may impact on core symptoms (at least for young and/or non-verbal people on the autism spectrum). Whether also this type of machine-analysis is applicable to other schedules related to autism screening and assessment is another question.

* Wall DP. et al. Use of machine learning to shorten observation-based screening and diagnosis of autism. Translational Psychiatry. April 2012.
DOI: 10.1038/tp.2012.10