Post from the ME-list 5/06/08, with links and annotation.
********************************************************
"Representative model programs were selected for the purpose of illustrating key features related to program effectiveness; however, this is not an exhaustive review, and not all existing programs are described here."
The quality of the data from each of the ten programs is described here in order to demonstrate that the vast majority of models do not have adequate data to be considered effective. Unlike the groundbreaking New York State Department of Health Report, which used stringent criteria in determining which data to
include in the study, the National Academy of Sciences paper seems to be a free-for-all; any and all autism treatment models are allowed in, despite the glaring lack of peer-reviewed, scientific treatment data.
(Romanczyk et al., 2000).
Even according to the National Academy of Sciences, this model has no peer-reviewed outcome
data. The report states:
“outcomes have been reported primarily in non-peer-reviewed book chapters” p. 167.
What is not stated here is that the book chapter was in a book authored by Harris, one of
the members on the NAS Committee. Again, this builds the case that the NAS report is flawed by a conflict of interest on the part of the authors of this research.
Moreover, if there is no outcome data, then this model should not even be considered by any reputable group of academics. Put simply, in terms of legitimate science, the results of this ‘research’ did not even occur.
(Rogers et al., 2000).
This is the Sally Rogers model of autism treatment; Professor Rogers is one of the members of the NAS committee. They report four peer-reviewed outcome reports on this treatment model. However, when they use a control group, it is not a group of children with autism; rather, it is a group of children with other disorders of behavior and development.
According to Dr. Lovaas, the I.Q. gains of Rogers, when converted to ratio I.Q. scores, are quite small, ranging from three to nine points for the 10 reported, pre and post-treatment comparisons (Lovaas, 2003: 392). In addition, Rogers et al., report gains in language that are hypothesized to be greater than without treatment; however, this is using a predictive model that assumes a slower rate of language acquisition if there is no treatment. These are predictive statements and not based on a control group of children that are not receiving treatment.
KEY POINT: Due to a lack of equivalent control group, we do not know whether the supposed gains in this model are based on the treatment model itself or possibly on other variables, which may co-vary. In order to make any inferences regarding the efficacy of the Denver model, there must be a control group of children with autism receiving a different form of treatment and then the models could be properly compared.
(Harris et al., 2000).
Harris, S.L., J.S.Handleman, M.S.Arnold, and R.F.Gordon 2000 The Douglass Developmental Disabilities Center: Two models of service delivery. Pp. 233–260 in Preschool Education Programs for Children with Autism (2nd ed.), J.S.Handleman and S.L.Harris, eds. Austin, TX: Pro-Ed.
This is Sandra Harris’ model — also a member of the NAS committee that authored the report. The NAS report on the peer-reviewed studies of the DDDC program describing outcomes of twenty-seven children who went through their program. Although the measures taken are legitimate (Stanford-Binet and the Vineland I.Q. tests) once again, there is no control group with which to compare outcome.
It is not surprising, however, that the DDDC data reports some gains in I.Q. since the DDDC relies heavily on discrete trial training (a Lovaas style technique) to introduce new material, and support is also provided to parents for home-based treatment programming (Harris et al., 2000: 282). In fact, these results could be expected, due to the similarities with Lovaas’ autism treatment research. However, without a control group, the DDDC results are only useful for science if we look at them as a loose replication of Lovaas’ studies.
(Greenspan and Wieder, 1999).
Greenspan, S.I., and S.Wieder 1997 Developmental patterns and outcomes in infants and children with disorders in relating and communicating: A chart review of 200 cases of children with autistic spectrum diagnoses. The Journal of Developmental and Learning Disorders 1:87–141.
This model, developed by Greenspan and Wieder, has no peer-reviewed data. According to the NAS, the children’s outcome is based on “clinical notes … scores on the Functional
Emotional Assessment Scale,” and “ratings were also not blind to intervention status.” In addition, it’s important to focus on the important fact that there are no objective tests given by psychologists not invested in the study.
No I.Q. or adaptive functioning measures were given. In addition, their results were published in a book written by the authors — not in a peer-reviewed journal. In terms of genuine science, these autism treatment results did not occur.
(Dunlap and Fox, 1999a, 1999b).
1999a A demonstration of behavioral support for young children with autism. Journal of Positive Behavioral Interventions 2:77–87.
1999b Supporting families of young children with autism. Infants and Young Children 12:48–54.
This model is problematic for several reasons. This comes out of the Positive Behavioral Support literature (Dunlap), in which the goal is not the amelioration of autism. Their goal is support, not treatment. In addition, there is only experimental data reported on one (1) child. Perhaps the therapeutic criterion is met for that single child, focusing on the single, specific problem the child may have; however, the experimental criterion is not met without a control group. In addition, generalization from one child to the population of children with autism at large is impossible. We do not know how representative that child was to the population of children with autism at large. In addition, without a control group, there are too many variables that may have created the results reported. Moreover, the measures that were used are not sufficiently rigorous. There were no commonly accepted I.Q. measures taken i.e., Stanford-Binet, Vineland, and there was no comprehensive before and after measurements taken of a variety of variables such as adaptive behavior, or degree of self-stimulatory behavior.
(Strain and Cordisco, 1994; Strain and Hoyson, 2000).
This is a model with which Odom (yet another member of the NAS committee), is associated. The LEAP preschool children were followed from entry into the program, to exit (two years later) and then at age ten. The progress of these children cannot be attributed to the program itself because there was no control group with which to compare results. In addition, the only legitimate I.Q. test given was the Stanford-Binet, which was not given at intake, but
rather only after the children had left the program. Therefore, no conclusions as to the efficacy of the preschool program can be made since (a) we do not have the pretest scores, and (b) there was no control group, which is essential to control for possible confounding variables such as maturation of the children over the two year study period.
(Koegel et al., 1998).
Koegel, L.K., S.M.Camarata, M.Valdez-Menchaca, and R.L.Koegel 1998 Setting generalization of question-asking by children with autism. American Journal on Mental Retardation 102:346–357.
The pivotal response model bases its conclusions on the data of ten children; however, even according to the NAS, we can conclude the following:
“the absence of experimental design leaves it unclear whether these improvements can be attributed directly to the program’s interventions.” p. 169.
This research is attempting to discern whether social communicative initiation is a predictor of better outcome or a “pivotal behavior in autism treatment”. Even if they could show that this is a predictor of better outcome, they have not shown that their model teaches these children to initiate ‘better’ than any other autism treatment model since in the absence of an experimental a control group, no legitimate comparison can be made.
(Watson et al., 1989; Marcus et al., 2000).
Marcus, L., E.Schopler, and C.Lord 2000 TEACCH services for preschool children. In Preschool Education Programs for Children with Autism, J.S.Handleman and S.L.Harris, eds. Austin, TX: Pro-Ed.
Watson, L., C.Lord, B.Schaffer, and E.Schopler 1989 Teaching Spontaneous Communication to Autistic and Developmentally Handicapped Children. New York: Irvington Press.
According to the NAS:
“these studies are not direct evaluations of treatment outcomes” p. 170.
The only data this study shows is that when children in discrete trial classrooms also get an additional ten hours at home as well, then they benefit. This, however, does not show that the TEACCH program in itself is effective — it may suggest that more hours of treatment are effective. This study ideally should have compared a group that had ten more hours of discrete-trial, at home, with a group that received ten hours of TEACCH at home in order to discern if the TEACCH model of autism treatment is more effective. In short, in terms of the generally accepted principles of science, there are no results that support the efficacy of the TEACCH protocol in autism treatment.
(Smith et al., 2000a).
Smith, T., P.A.Donahoe, and B.J.Davis 2000a The UCLA Young Autism Project. Pp. 29–48 in Preschool Education Programs for Children with Autism. J.S.Handleman and S.L.Harris, eds. Austin, TX: Pro-Ed.
Despite their acknowledgement of the attacks on Lovaas’ research by Gresham et al., the
NAS admits this:
“the UCLA program has generated the most rigorously controlled early intervention research published to date … the Young Autism project has also reported the longest follow-up tracking of children with autism who have received intensive early intervention” p. 170.
They also refer to five of the replications of the UCLA model.
Points to remember:
1) Lovaas used one experimental group, and two control groups in his original study
2) He has a longitudinal study looking at whether the gains of the children were maintained over a decade later
There are no less than six published peer-reviewed studies in which EVERY experiment group significantly improved over the control group.
These studies are:
i. Lovaas (1987) and McEachin et al. (1993),
ii. Birnbrauer and Leach (1993),
iii. Smith et al. (1997),
iv. Sheinkopf and Siegel (1998),
v. Smith (2000),
vi. Eikeseth et al. (2002)
(there is also one additional replication in press, Graupner and Sallows).
(McGee et al., 2000).
2000 The Walden Early Childhood Programs. Pp. 157–190 in Preschool Education Programs for Children with Autism, 2nd ed. J.S.Handleman and S.L.Harris, eds. Austin, TX: Pro-Ed.
The NAS correctly points out that with the exception of the Toddler Program, data on the Walden program is not in peer-reviewed journals. Even the Toddler Program, which is peer-reviewed, does not have “experimental controls to provide the majority of the data.” (P.17). In other words, we cannot attribute the toddler’s gains in the areas of social readiness and verbalizations to the pre-school program since there is all too familiar lack of a control group in the study — there is no way to legitimately compare any gains.
Unfortunately, without experimental controls, the data is meaningless, particularly when it comes to language acquisition for children between the ages of 2-1/2 and 3 years of age — the time when children are most likely to acquire language. This is especially true when we don’t have standardized tests that can be compared to the language development in typically developing children. To report increased verbalizations, without being able to measure the extent of the gains and then compare them to typical peers, as opposed to a control group of autistic children, really tells us nothing about the effect of the Walden Toddler program on these autistic children. In addition, once these toddlers go from the toddler program into the preschool program, there is no peer-reviewed outcome data on the efficacy of the Walden preschool program. Therefore, in terms of the generally accepted principles of science, these results did not occur.
Of the ten autism treatment models, four had no peer-reviewed data, and five have no control group. Only one model has peer-reviewed data, longitudinal studies, and four replications. Yet the NAS authors have no problems weighing all these programs equally, looking at common elements that they believe are effective. Despite their supposed expertise in science, the NAS authors of this report were inexplicably unable to conclude from their own analysis that only one of these treatment programs — the UCLA model — clearly has the best outcome data, both in quality of science and efficacy of treatment.
I. Children’s Unit – NO PEER-REVIEWED DATA
II. Denver Model– NO EQUIVALENT CONTROL GROUP
III. Douglass Developmental Disabilities Center – NO CONTROL GROUP
IV. Developmental Intervention Model – NO PEER REVIEWED DATA
V. Individualized Support Program – NO CONTROL GROUP, SMALL (N=1)
IV. Developmental Intervention Model – NO PEER REVIEWED DATA
VII. Pivotal Response Model –NO CONTROL GROUP
VIII. TEACCH – NO PEER-REVIEWED OUTCOME DATA
IX. UCLA Young Autism Project – 1 EXPERIMENTAL GROUP, 2 CONTROL GROUPS, LONGITUDINAL DATA, 6 PEER-REVIEWED OUTCOME STUDIES WITH DATA
X.Walden Toddler Program – NO CONTROL GROUP
An important difference between the New York State Department of Health Report and the NAS Report is this: NONE of the models discussed in this analysis (with the exception of Lovaas and replications) was analyzed in New York because they simply did not have the data and experimental designs required to meet even the minimum standards of eligibility to be even considered for review in New York State Department of Health Report. The argument that the New York Report was published in 1999 and that the state of the science has advanced since then, can be easily countered by looking at the dates of the articles in journals with peer-reviewed data regarding the above models: Rogers, 1988; 1991; 1987; 1986; Strain 1984; 1987 (Strain 2000 uses children from earlier than 1992), Greenspan 1997; Harris et al. 1990; 1991; 1995;McGee, 1993; 1994; 1999, Dunlap et al.; 1999, Ozonoff; Cathcart, 1998.
Under Intensive Behavioral and Educational Intervention Programs in the New York Report, only the following articles had sufficient data and competent experimental design to be included:
• Lovaas (1987)
• McEachin et al. (1993)
• Birnbrauer and Leach (1993)
• Smith et al. (1997)
• Sheinkopf and Siegel (1998)
With respect to Greenspan et al, 1997, the New York Report states the following:
“However, there is currently no adequate scientific evidence (based on controlled studies using generally accepted scientific methodology) that demonstrates the effectiveness of DIR-based interventions for young children with autism. Therefore, the use of these approaches cannot be recommended as a primary intervention method for young children with autism. Furthermore, it is important to recognize that approaches based on the DIR model can be time-intensive for both professionals and parents and may take time away from other therapies
that have been demonstrated to be effective” p. IV-58.
The fact that the NAS did not arrive the same, or similar, conclusions as the New York State Department of Health Report, makes the inevitable argument that the NAS authors have an agenda of their own. This may be to blunt conclusions of the New York report, by arguing that when it comes to autism treatment, any given method is apparently no more effective than any another protocol.
Notable is that the New York Report used research standards based on the Agency for Health Care Policy and Research (AHCPR) which is a part of the United States Public Health Service and considered the industry standard in the development of clinical practice guidelines. In other words, the New York Report on autism treatment is as rigorous as it gets in the science business.
Note:
In terms of medical treatment for children with autism, it is telling that the U.S. Surgeon General, in the Report on Mental Health (1999), praised Lovaas’ design and outcomes as being impressive, but did not even mention the vast majority of the models listed in the NAS paper. Clearly the U.S. Surgeon General considered Lovaas’ protocol as medically necessary medical treatment, not simply an educational program.