Comparing Distance Education and Traditional Instruction

Comparing Distance Education and Traditional Instruction:
An "In-progress" Meta-Analysis of the Empirical Literature
Robert M. Bernard
Professor
Concordia University
Montreal, Quebec Yiping Lou
Assistant Professor
Louisiana State University
Baton Rouge, LA Philip C. Abrami
Professor
Concordia University
Montreal, Quebec
The authors gratefully acknowledge the assistance of Lori Wozney, Binru Huang, Peter Wallet, Evgueni Borokhouski and Manon Fiset, all research assistants. This research is supported by a grant from the Social Sciences Research Council of Canada.
Introduction
The research that is reported herein impinges upon the following basic question: Should respectable educational institutions continue to develop and market Internet and distance education learning opportunities without knowing whether they will be as effective as their classroom-based equivalents, or in the worse case, whether they will be effective at all? Based on longstanding instructional design thinking, it is not enough to develop a technologically-based course simply because the medium of delivery exists, and yet the reverse of this very thinking seems to prevail in the rush to get courses online.
Yes, there have been attempts to summarize the comparative research literature of DE. The most comprehensive is Russell’s (1999) collection of 355 (and counting) "no significant difference" studies (see also Phipps & Merisotis, 1999). Contrasting this number with the number of "significant difference studies" (i.e., the box-score method of review), Russell declares there is compelling evidence to support Richard Clark’s (1983) original claim that a delivery medium contributes little if anything to the outcomes of planned instruction—in other words, that distance can be no better than traditional instruction. But as we all know, differential sample sizes of individual studies makes it impossible to aggregate them solely on the basis of their test statistics. So Russell’s approach fails to answer the difference question and provides no help to educators and policy-makers in need of research to guide their decision-making. Other more limited attempts have been made to summarize the literature both quantitatively and qualitatively (Waight, Willging & Wentling, 2002; Berge & Mrozowski, 2001; Cavanaugh, 2001; Machtmes, 1999; Schlosser & Anderson, 1994; McNeil & Nelson, 1991), but none of these has adequately addressed the myriad of questions, both substantive and methodological, that arise from this large and complex research literature.
The primary answer to those who contend that this line of comparative research has yielded little more than non-significant findings is that "no significant difference" does not "prove the null" and that a comprehensive meta-analysis is required. Meta-analysis or quantitative synthesis was developed by Glass, et al. (1981) to avoid the very problems that are so evident in Russell’s work. Effect size, the metric of a meta-analysis, is an unbiased index of standardized differences between a treatment and control group, and can be averaged in a way that test statistics cannot.
However, there is still the thorny issue of media/method confounds to be dealt with, the main criticism raised by Clark (1983, 1994), Bernard (1986) and others regarding studies of technology. Several authors have suggested how this conundrum might be addressed from different perspectives (e.g., Smith & Dillon, 1999; Cobb, 1997; Kozma, 1994; Morrison, 1994; Ullmer, 1994). For the purposes of the current research, the main argument for recognizing and extending this kind of research is the fact that when constructs of delivery system, media and instructional potential are clearly defined and combined, subtle differences in the effect of two different approaches on learners and learning may emerge. The overall purpose of our research, therefore, is to synthesize empirical research on the effects of DE using the techniques of meta-analysis and to explore it by testing the effects of various study features, both alone and in combination (Hedges & Olkin, 1985).
In general, this meta-analysis seeks to answer the following questions: 1) Does DE compared to traditional courses improve student achievement and other outcomes related to attitude and
success? If so, to what extent? 2) What study features moderate the effects of DE on student achievement and other outcomes? 3) What are some optimal conditions for effective distance learning and what causes it to fail?
Data Sources and Inclusion/Exclusion Criteria
Our working definition of DE builds on Nipper‘s (1989) and Taylor’s (2001) models of "third generation plus" distance learning, as well as Keegan‘s (1996) synthesis of recent definitions. Linked historically to developments in technology, first generation DE refers to the early days of print-based correspondence study. Characterized by the establishment of the Open University in 1969, second generation DE refers to the period when print materials were integrated with broadcast TV and radio, audio and video cassettes and increased student support. Third generation DE was heralded by the invention of Hypertext and the rise in the use of teleconferencing (i.e., audio and video). To this, Taylor (2001) adds the fourth generation, characterized by flexible learning (e.g., CMC, Internet accessible courses) and the fifth generation (e.g., interactive multimedia online, Internet-based access to WWW resources). Generations three, four and five represent moves away from authoritarian and non-interactive courses towards those characterized by a high degree of learner control and two-way communication, as well as group-oriented processes and greater flexibility in learning. With new communication technologies in hand and renewed interest in the convergence of DE and traditional education, this is an appropriate time to review the research on third, fourth and fifth generation DE. Our definition of DE for the inclusion of studies thus reads:
The semi-permanent separation (place and/or time) of learner and instructor during planned learning events. The influence of the educational organization on the planning and preparation of the learning materials, student support services and the final recognition of course completion. The provision of two-way media to facilitate dialogue and interaction between the students and the instructor and allow for temporal control of this interaction.
The studies were located through comprehensive literature searches on ERIC, PsycInfo, Educational Technology Abstracts, Social SciSearch, Dissertation Abstracts, Education Abstracts, ABI, CBCA Education, Canadian Research Index, Applied Science and Technology Index, Cambridge Scientific Abstracts and the reference lists of several reviews. Additional manual searches were also performed. The search terms used were "distance education," "distance learning," virtual university" and "comparative study." To be included in this review, each study also met eight criteria for inclusion/exclusion, the following being the most important:
Each study must involve an empirical comparison of DE with face-to-face classroom instruction (including lectures, seminars, tutorials and laboratory sessions). Statistical data for calculating effect sizes must be present.
Method and Procedures
Development of the codebook was initiated from a review of a sample of 10 retrieved studies, as well as review pieces (e.g., Phipps & Merisotis, 1999), conceptual papers (e.g., Smith & Dillon, 1999) and critiques (e.g., Saba, 1999). To allow for a direct comparison of DE and face-to-face instructional conditions, many of the study features were written in the following form:
DE more than control group DE reported/control group not reported DE equal to control condition Control reported/DE not reported DE less than control group
999. Missing (no information on DE or control reported)
A sample of studies were first nomologically coded to identify salient study features present in the literature and to avoid researcher bias (Abrami, et al., 1988). Through nomological coding, a comprehensive codebook was developed, which includes the following study feature categories: research design; outcome measures (i.e., achievement, success rate, attitude towards course); accessibility (e.g., technical support); media used (e.g., attributes); human factors (e.g., learner/instructor differences); course design (e.g., learning tasks); and pedagogy (e.g., collaboration). Of particular interest in our analysis are the related constructs of one-way versus two-way communication, interactivity and students’ active engagement. Synchronous versus asynchronous communication and the important issue of time on task is also considered, together with features related to instructional control and group interaction.
After identifying studies for potential inclusion, two research assistants judged them for inclusion using the inclusion/exclusion criteria previously described. The interrater agreement was 90.7%. One hundred and sixty-nine (169) manuscripts were selected for inclusion in the meta-analysis.
Next, effect sizes were extracted using the procedures outlined by Glass, et al. (1981) and Hedges, et al. (1989). Each study was coded by two research assistants separately and then compared for reliability. The interrater agreement on effect size coding and calculation was 85%.
In the final stage to come, findings from all included studies will be integrated and analyzed using homogeneity procedures. If the overall mean effect sizes are significantly heterogeneous, variability in the findings will be explored through study feature analyses to develop a model of factors that are significantly related to the effects of interactive DE that is as parsimonious as possible.
Results to Date
A total of 712 empirical studies, dated between 1985 and July 2002, have been retrieved and 169 of these have been selected for inclusion. To date, 160 studies have been coded for effect size. The results reported below are based upon 344 independent effect sizes from 137 studies representing 93,416 students in DE and traditional classes.
Measure
N1
Mean E.S.2
95% C.I.3
Q4 Statistic
Achievement
155
+.24
+.22 to +.26
4558.35, p < .001
Success Rate
26
–.11
–.06 to –.05
73.85, p < .001
Attitude: Course
60
+.03
+.06 to +.15
411.49, p < .001
 
1 Number of effect sizes analyzed. (Note: three measures and missing data are not reported.)
2 Mean Effect Sizes are Cohen’s d (where appropriate, standard deviations are pooled) and are weighted for differential sample sizes.
3 95% confidence intervals. A Mean ES lying within this range is significant at the .05 level.
4 Q is a measure of variability and p is the probability associated with the test of Homogeneity of Effect Size (e.g., the range of achievement was from –.17 to +3.5).
These results suggest that, on average, students in interactive DE achieved a modest amount more (i.e., roughly 10%) on measures of achievement than those in traditional classrooms. Success rate, however, was negative, as the literature of DE suggests, but not significant. The effect size for "attitude towards course" was positive but essentially zero and not significant. All of the average effect sizes violated homogeneity of effect size. This variability within the various findings will be explored using the study features coded to identify significant moderating factors on the effects of DE on achievement, success rate and attitude.
Significance and Contribution
This meta-analysis will provide a better understanding of the effects of DE on student achievement and other outcomes by answering questions about the extent to which DE is more or less effective than traditional instruction. When completed, it may also suggest the conditions under which the application of DE is effective or ineffective. With the increasing power of communication technologies and the need for lifelong learning, DE will play an increasingly important role in education. The results from this meta-analysis will provide policy-makers as well as educators with important guidance, based on past research, on how best to proceed in the future development of distance education.
References
Abrami, P.C., P. Cohen, P., & d‘Apollonia, S. (1988). Implementation problems in meta-analysis. Review of Educational Research, 58(2), 151-179.
Berge, Z. L., & Mrozowski, S. (2001). Review of research in distance education, 1990 to 1999. The American Journal of Distance Education, 15(3).
Bernard, R. M. (1986). Is research in new technology caught in the same old trap? Canadian Journal of Educational Communication, 15(3), 147-151.
Cavanaugh, C. S. (2001). The effectiveness of interactive distance education technologies in K-12 learning: A meta-analysis. International Journal of Educational Telecommunications, 7(1), 73-88. Available online: http://www.unf.edu/~ccavanau/CavanaughIJET01.pdf
Clark, R.E. (1983). Reconsidering research on learning from media. Review of Educational Research, 53(4), 445-459.
Clark, R.E. (1994). Media will never influence learning. Educational Technology Research & Development, 42(2), 21-29.
Cobb, T. (1997). Cognitive efficiency: Toward a revised theory of media. Educational Technology Research & Development, 45(4), 21-35.
Glass, G. V., McGaw, B., & Smith, M. L. (1981). Meta-analysis in social research. Beverly Hills, CA: Sage.
Hedges, L. V. & Olkin, I. (1985). Statistical methods for meta-analysis. Orlando, FL: Academic Press.
Hedges, L.V., Shymansky, J.A., & Woodworth, G. (1989). A practical guide to modern methods of meta-analysis. (ERIC Document Reproduction Service No. ED 309 952).
Keegan, D. (1996). Foundations of Distance Education. (3rd Edition) (Ch. 3: Definition of Distance Education). London: Routledge.
Kozma, R.B. (1994). Will media influence learning? Reframing the debate. Educational Technology Research & Development, 42(2), 7-19.
Machtmes, K. & Asher, J.W. (2000). A meta-analysis of the effectiveness of telecourses in distance education. The American Journal of Distance Education. 14(1): 27-46.
McNeil, B. J., & Nelson, K. R. (1991). Meta-analysis of interactive video instruction: A ten-year review of achievement effects. Journal of Computer-Based Instruction, 18(1).
Morrison, G. R. (1994). The media effects question: Unresolveable or asking the right question. Educational Technology Research & Development, 42,(2), 41-44.
Nipper, S. (1989). Third generation distance learning and computer conferencing. In: R. Mason and A. Kaye (eds), Mindweave Communication, Computers and Distance Education, (pp. 63-73), Oxford, UK: Pergamon Press.
Phipps, R. & Merisotis, J. (1999). What’s the difference? A review of contemporary research on the effectiveness of distance learning in higher education. Washington, DC: The Institute for Higher Education Policy.
Russell, T.L. (1999). The no significant difference phenomenon. Chapel Hill, NC: Office of Instructional Telecommunications, North Carolina State University.
Saba, F. (1999). Toward a systems theory of distance education. The American Journal of Distance Education. 13(2):24-31.
Schlosser, C. A., & Anderson, M. L. (1994). Distance education: Review of the literature. Unpublished doctoral dissertation, Iowa State University; Ames, IA.
Smith, P.L. & Dillon, C.L. (1999). Comparing distance learning and classroom learning: Conceptual considerations. The American Journal of Distance Education, 13(2), 107-124.
Taylor, J.C. (2001). Fifth generation distance education. Keynote address
delivered at the ICDE 20th World Conference, Dusseldorf, Germany, 1-5 April. Available on-line at: http://www.usq.edu.au/users/taylorj/conferences.htm
Ullmer, E.J. (1994). Media and learning: Are there two kinds of truth? Educational Technology Research & Development, 42(2), 21-32.
Waight, C. L., Willging, P. A., & Wentling, T. L. (2002). Recurrent themes in e-learning: A meta-analysis of major e-learning reports. Available on-line at: http://www.learning.ncsa.uiuc.edu/papers/ AHRD2002_waight-willging-wentling.pdf