Canadian Journal of Learning and Technology

Volume 31(2) Winter / hiver 2005

Exploring Instructional Design Factors Prompting Reflective Thinking in Young Adolescents

Hae-Deok Song

Tiffany A. Koszalka

Barbara L. Grabowski

Authors

Hae-Deok Song is an Assistant Professor in the faculty of Educational Theory and Practice at the University of Albany. Correspondence concerning this article can be sent to hsong@uamail.albany.edu

Tiffany A. Koszalka is an Associate Professor of Instructional Design, Development and Evaluation at Syracuse University. She can be contacted at takoszal@syr.edu

Barbara Grabowski is a Professor of Education at Penn State University. She can be contacted at blg104@psu.edu

Abstract

Abstract: Reflective thinking is important to young adolescents as they develop their thinking skills. Various instructional methods have been recommended to support reflective thinking, yet the nature of the underlying factors in these methods is unclear. Exploratory factor analysis was used to determine the factors prompting reflective thinking. Results of this study suggest that young students perceived three clusters of methods as supporting their reflection: reflective learning environments, reflective teaching methods, and reflective scaffolding tools. A one-way within subjects ANOVA showed that the most helpful factor was the reflective learning environment, with the most helpful elements being freedom and collaboration. Students’ perceptions of concept mapping and reflective question prompts were found to differ significantly across gender. Recommendations are provided for designing learning environments that prompt reflective thinking.

Résumé: La réflexion est importante chez les jeunes adolescents tandis qu’ils perfectionnent leurs aptitudes en matière de réflexion. Diverses méthodes d’enseignement ont été recommandées pour appuyer la réflexion, toutefois, la nature des facteurs sous-jacents de ces méthodes est incertaine. On a utilisé l’analyse du facteur exploratoire pour identifier les facteurs menant à la réflexion. Les résultats de l’étude démontrent que les jeunes étudiants perçoivent trois groupes de méthodes qui appuient leur réflexion: les milieux d’apprentissage réfléchi, les méthodes d’enseignement réfléchi et les outils d’appui réfléchi. L’analyse de variance ANOVA à un facteur a démontré que le facteur le plus utile était l’environnement d’apprentissage réfléchi, les éléments les plus utiles étant la liberté et la collaboration. Les perceptions des étudiants relatives aux questionnements sur le schéma conceptuel et le questionnement réfléchi semblent varier de façon importante d’un sexe à l’autre. Des recommandations sont faites relativement à la conception d’environnements d’apprentissage qui incitent à la réflexion.

Introduction

Reflective thinking, that is, mentally engaging in cognitive processes to understand conflicting factors in a situation, is a critical component of the learning process (Atkins & Murphy, 1993; Boyd & Fales, 1983; Davis, 1998; Dewey, 1933; Moon, 1999; Schön, 1991). This mental engagement results in a person’s actively constructing knowledge about a situation in order to develop a strategy to proceed within that situation. For example, after a series of instructional lessons on ecology and waste management, biomes, life cycles and urban development, students may be given the task of determining whether a parcel of land should be protected and then justifying this position. Students must reflect on their previous understanding of the issue and their newly acquired knowledge in order to respond to an issue. In effect, they must think about the situation and reflect on how their new knowledge can be used to address the situation, ultimately developing a solution to the land-protection issue. Thus, reflection helps students to develop higher-order thinking skills by prompting them to: (a) relate new knowledge to their prior understanding, (b) think in both abstract and concrete terms, (c) apply specific strategies to novel tasks, and (d) understand their own thinking and learning strategies (Hmelo & Ferrari, 1997).

Key Elements Prompting Reflective Thinking

Young adolescents in the period of developmental transition need support to think reflectively (Lin, Hmelo, Kinzer, & Secules, 1999; National Middle School Association, 1995). Without such support, they have difficulty relating new information to prior knowledge or choosing specific strategies in new situations. Previous research on reflective thinking suggests that three areas are particularly important in prompting reflective thinking, although various elements can do so. First, particular teaching methods can influence the development of students’ reflective-thinking skills (Virtanen, Kosunen, Holmberg-Marttila, & Virjo, 1999). Teachers who prefer inquiry-oriented activities help students to reflect on a situation by asking thoughtful questions. Explanation-oriented teachers describe concepts to students, thereby prompting reflective thinking based on newly presented information (Moon, 1999; Virtanen, Kosunen, Holmberg-Marttila, & Virjo, 1999). Teachers who also use wait-time effectively can prompt students to think reflectively before they react (Rowe, 1974). The authenticity and real-life experience that the teacher brings into classroom learning activities also helps students think reflectively by providing real situations and contextualized knowledge about new information that they are learning. Next, scaffolding tools such as interactive journals, question prompts, and concept maps, also prompt reflective thinking (Griffith & Frieden, 2000; Kinchin & Hay, 2000). For example, Andrusyszyn and Daive (1997) noted that students who participated in interactive journal writing perceived the journal as a tool that helped them think about personal experiences, summarize their learning, and share their learning experiences with others. Finally, the learning environment may prompt students to construct meaning actively and reflectively. Providing learner-controlled instruction encourages students to make their own decisions regarding their learning progress (Duffy & Cunningham, 1996; Williams, 1996). Collaborative learning, in which students explore their understandings and misunderstandings together, helps students to think about what they already know, what they need to know, and how they would present and defend their own ideas in reaction to an instructional situation (Aldred & Aldred, 1998). Complex learning activities requiring students to learn from multiple forms of information before responding tend to elicit active consideration of multiple inputs when students make judgments on how to proceed (Stepien & Pyke, 1997).

Although it is widely believed that incorporating elements from the three areas mentioned above in a learning context prompts young adolescents’ reflective thinking, it is not clear how these elements are related to each other and which elements are essential in prompting and enhancing students’ reflective-thinking skills.

Exploring Some Factors that Prompt Reflective Thinking through Students?Perceptions

It is important to take a mindful stance toward the development of learning environments that include scaffolds to prompt reflective thinking and ultimately knowledge construction. As noted previously, numerous studies have discussed elements that prompt reflective thinking; however, the constructs underlying these methods are not clear. To develop meaningful instructional design principles, it is important to investigate empirically the fundamental factors within which various elements can be categorized (Klimczak & Wedman, 1997).

Despite the plentiful suggestions for methods to help students develop their reflective-thinking skills, few researchers have attempted to examine students’ perceptions of the helpfulness of those methods (Malpine & Weston, 2000) . One step in identifying empirically driven instructional design recommendations is to understand students’ perceptions of the ability of these design elements to help them to reflect, because this is not clear. Even when the instructional methods and strategies that teachers use are known to be effective, instruction may not be understood or perceived well by students (Wittrock, 1990). Students may not perceive these designed environmental elements as helpful in prompting reflection; or they may perceive other environmental elements as more helpful. In both cases, the students’ perceptions of instructional elements may affect their learning experience. Thus, understanding middle-school students’ perceptions can inform the practice of developing learning environments that encourage reflection, and ultimately result in more effective learning.

Further investigation of student perceptions by individual differences is also needed, to discover which underlying factors are meaningful. Students may perceive the effectiveness of particular design factors differently according to their individual characteristics (Gallagher, 1994; Roeser, Eccles, & Sameroff, 2000). Gender, age and grade level have been investigated extensively as crucial individual differences over the past few decades (Butler & Manning, 1998; Craig, 1983; Wolman, 1982). These factors are also examined in this study.

Purpose of the Study

The purpose of this study was to explore some of the factors that theoretically prompt reflective thinking in middle-school students by examining their perceptions of the ability of various instructional design elements to prompt their reflection. The following research questions guided the study:

1. Which instructional design elements load together as factors that middle-school students perceive as helpful in prompting their reflective thinking?

2. Which of the identified factors and elements are perceived as the most helpful in prompting their reflective thinking?

3. Are there differences in students’ perception of the elements based on grade level or gender?

Methods

Participants and Procedures

The participants in this study consisted of 141 students from three middle schools located in a north-eastern state. Participants included 82 boys and 59 girls. Participants came from intact groups in six different classrooms representing the 6 th (n=84), 7 th (n=39), and 8 th (n=18) grades.

A ten-item survey was administered at the beginning of each group’s science class. The survey was designed to measure middle-school students’ perceptions of the underlying factors that supposedly prompt reflective thinking. The survey was administered by the students’ science classroom teachers. Since students’ perceptions could be affected by their participation in different science classroom learning environments, the survey was administered at the beginning of the semester, before classes began. This procedure enabled us to better control for environmental variables.

Measurement Instrument

A survey developed by the researcher was used because no pre-existing instrument was available to measure the perceived helpfulness of factors prompting reflective thinking (Koszalka & Song, 2002). The survey instrument was composed of ten carefully targeted question items based on an extensive examination of the reflective-thinking literature. Each item on the survey queried students about one instructional design element suggested by the literature. The design elements included teacher explanations and questions, real-life tasks and wait-time, the use of scaffolding tools such as concept mapping, reflective writing and reflective question prompts, and supportive and flexible learning-environment elements including collaborative learning, complex learning activities and learner control (Aldred & Aldred, 1998; Andrusyszyn & Daive, 1997; Barrows, 1998; Griffith & Frieden, 2000; Kinchin & Hay, 2000; Moon, 1999; Rowe, 1974; Stepien & Pyke, 1997; Virtanen, Kosunen, Holmberg-Marttila, & Virjo, 1999). Students were asked to respond to each statement using a five-point Likert scale ranging from strongly agree (5) to strongly disagree (1).

In an effort to ensure content validity, faculty from a major university in a north-eastern state, with expertise in the area of reflective thinking, were asked to review the questions and assess the potential of the factors for prompt reflective thinking. All agreed that the items represented important elements hypothesized to prompt reflective thinking. Therefore, all items were retained.

In an effort to ensure readability and consistent interpretation by the intended audience, the survey instrument was pilot-tested with a sample of 3 children in the sixth, seventh, and eighth grades (Koszalka, Song, & Grabowski, 2002). Each was asked to review and interpret the initial draft of the questions. This initial review confirmed that some terminology was too difficult for younger children. The items were revised based on this feedback, and a second sample of 5 different sixth grade students was asked to review the new version. Again, the items were edited in response to their feedback. For example, “working with complex tasks” was changed to “working on activities in class that have many different answers.” After these students had completed the survey, 5 new students from each of the three groups (6 th, 7 th, and 8 th-grades) were interviewed to assess their understanding of the meaning of each item. The researchers asked the students to explain how helpful they thought each item was in prompting their thinking, and they paraphrased the meaning of each item. Their verbal responses were compared with their written survey responses. The data collected during the student interviews indicated that students understood the items and answered them consistently on the written survey and in the oral interviews.

Data Analysis

Exploratory factor analyses (EFA) employing principal component analysis with the varimax rotation method were used to determine the factors prompting reflective thinking. We used EFA instead of employing confirmatory factor analysis because the factor analysis in this study was meant primarily not to test an established factor structure but to develop a factor structure classifying the elements prompting reflective thinking (Tabachnick & Fidel, 2001). Further analyses of the data were conducted to examine the characteristics of the instructional design factors that emerged from the EFA. First, a one-way within-subjects ANOVA was run to identify the most highly regarded factor and element. Within-subjects analysis was run since all subjects responded to all the items in the measurement instrument, and responses were compared to each other. Factor mean scores were used to conduct the ANOVA. Mean scores of each factor was computed by dividing the sum of item scores loaded to each factor by the number of items. Since a five-point Likert scale was used for each item on the survey, the calculated factor means scores have the same range and thus, the researchers were able to test the difference between factors that included the different items. The one-way within-subjects ANOVA was chosen instead of a paired-sample t-test in order to control the Type-I error rate, since the use of multiple t-tests increases the chance of making a Type-I error in the absence of any adjustment of the significance level. Second, a factorial MANOVA was conducted to test for differences based on grade level and gender among the items related to the factors prompting reflective thinking. It was conducted to ensure accurate Type-I error estimates. Tukey’s post hoc test for significant differences among groups was used as a follow-up test.

Results

Factor Structures from the Exploratory Factor Analysis

Before conducting the EFA, we carried out a preliminary analysis to find out whether the matrix used in this study was appropriate for the factor analyses. We examined the determinant of the correlation matrix to see if the variables correlated too highly. The value of the determinant was 0.064 greater than the necessary value of .00001, which indicated that multicollinearity was not a problem for the data used in this study. Kaiser-Meyer-Olkin (KMO)’s Measure of Sampling Adequacy (MSA) and a Bartlett Test of Sphericity were also conducted to determine the appropriateness of the data matrix. The KMO’s MSA measure varies between 0 and 1, and a value of 1 indicates that the pattern of correlation is relatively compact, so that the factor analysis should yield distinct and reliable factors. The MSA in this study was .806, which indicated a sound pattern of correlation. The Bartlett’s measure showed significant correlations among the variables in the correlation matrix: Χ² (45, N=141) = 373.21, p <.01. In sum, the preliminary analysis results supported the evidence that the factor analysis assumption for the matrix used in this study was tenable.

Principal component analysis with orthogonal (varimax) rotation was used to examine the factor structures of the ten items. Orthogonal rotation was chosen because the correlations between factors calculated from the component correlation matrix turned out to be relatively low (r < .30) and, thus, the researchers in this study could assume the independence of the factors. The number of factors extracted was based on three criteria: (a) the eigenvalue, (b) the percentage of variance, and (c) a scree test (Hair, Anderson, Tatham, & Black, 1995). The decision to retain an item was based on the following two criteria: (a) an item-structure coefficient greater than .40 and (b) a minimum gap of .10 between salient coefficients on multiple factors (Nunnally, 1978).

In response to the first research question, Which instructional design elements load together as factors that middle-school students perceive as helpful in prompting their reflective thinking?, three factors emerged with eigenvalues greater than 1, which together accounted for 60.6% of the variance in the data.

Four items loaded to factor 1: teacher explanations, teacher questions, teacher wait-time, and real-life tasks. Two items loaded to factor 2: reflective questions and reflective writing. Four items loaded to factor 3: collaborative learning, concept mapping, learner control, and complex learning activities. Table 1 presents the factor-structure matrix, eigenvalues, communalities, and variance for each factor.

Table 1. Factor Loading of Instructional Design Elements that Prompt Reflective Thinking

Factor 1 appears to relate most closely to reflective teaching methods. The four elements that loaded to the reflective teaching methods factor are all related to interventions that the teacher brings into the classroom. Factor 2 appears to relate to reflective scaffolding tools; reflective question prompts and writing are often used in the classroom to help students to develop reflective-thinking skills. Factor 3 appears to relate to reflective learning environments, defined by the data as having four elements: collaboration, learner-control, complex learning challenges and drawing pictures. The presence of Item 8, drawing pictures, in this group was unexpected; it was expected to load to reflective scaffoldingtools. From this data, it would seem that drawing pictures must have been perceived as helpful in the same way as the other more student-centered elements of this factor. Interestingly, Item 1, complex learning activities, was significant (.43), as expected, as part of the learning environment but also had a strong correlation (.40) with the scaffolding-tools factor and the gap between coefficients was not greater than 1.0. Because we focused on identifying constructs that lead to perceived helpfulness in terms of reflective thinking, we decided to retain Item 1.

To further analyze the data, we examined their internal-consistency reliability and bivariate correlations. Tables 2 and 3 show the means, standard deviations and correlation matrices for the three factors and ten items. The internal consistency for Factor 1 was .76; for Factor 2, .68, and for Factor 3, .61. The reliabilities of the three factors indicate that the variables making up each factor hang together fairly well.

Students?Perceptions of some Factors / Elements Prompting Reflective Thinking

In response to the second research question, Which of the identified factors was perceived as most helpful in prompting their reflective thinking?, Factor 3—the reflective learning environment emerged as the highest ranked factor, followed by Factor 1—reflective teaching methods, and Factor 2—reflective scaffolding tools (see Table 2). The one-way within-subjects ANOVA showed significant differences between the reflective learning environment factor (M = 3.86, SD = 0.66) and the reflective scaffolding tools factor (M = 3.22, SD = 0.93), F(1, 140) = 40.73, p <. 01, between the reflective teaching methods factor (M = 3.70, SD = 0.72) and the reflective scaffolding tools factor (M = 3.22, SD = 0.93), F(1, 140) = 64.40, p <. 01. The most significant factor, the reflective learning environment, included the student-centered attributes collaborative learning, learner control, concept-mapping activities and complex-learning activities. Thus, students perceived the student-centered characteristics of the learning environment as more helpful in prompting reflective thinking than the teacher-centered teaching methods and scaffolding tools.

The one-way within-subjects ANOVA also uncovered the significant elements within each factor. Of the items in the reflective learning environment factor, Item 7, learner control (M = 4.09, SD = 0.95) and Item 5, collaborative learning (M = 4.07, SD = 0.92) were perceived as the most helpful. These two items were not statistically different from each other, but were significantly higher than the other two items. For instance, collaborative learning came out significantly higher than the third highest ranked item, Item 1, complex learning activities (M = 3.71, SD = .93), F(1, 140) = 17.71, p <. 01. Of the items in the reflective teaching methods factor, Item 6, teacher wait-time (M = 3.92, SD = 0.81) was perceived as the most helpful. It was significantly higher than the second highest ranked item, Item 3, teacher explanation (M = 3.72, SD = 0.91), F(1, 140) = 6.92, p <. 05. Of the two items in the reflective scaffolding tools factor, Item 10, r eflective question prompts (M = 3.48, SD = 0.99) was perceived as the more helpful. It was significantly higher than the other item within the factor, Item 9, reflective writing (M = 2.96, SD = 1.13), F(1, 140) = 34.54, p <. 01.

Discussion

Six main conclusions can be drawn about middle-school students’ perceptions of the helpfulness of these three design factors and ten instructional design elements in prompting their thinking.

Three Factors Prompting Reflective Thinking

Three major factors emerged from the exploratory factor analysis. Students perceived reflective teaching methods, reflective scaffolding tools, and reflective learning environments as helpful in prompting their reflective thinking. Interestingly, two different learning-task elements found by Stepien and Pyke (1997), “authenticity represented as real-life tasks” and “complexity represented as complex learning activities”, loaded to both the reflective teaching methods and the reflective learning environment factors. Authenticity loaded to the reflective teaching methods factor. By including the authenticity element in this factor, the reflective teaching method factor can be defined by what the teacher brings to the situation, that is, bringing activities related to real problems. Complexity loaded to the reflective learning environments factor. By including complexity in this factor, the data suggest that the attributes of learning environment for middle-school students should include active involvement in complex tasks.

This may also explain a reason why students perceived concept mapping as similar to active and involved tasks, rather than as a scaffolding tool element as noted in the literature, although this finding was not consistent across grade levels, as will be discussed later. Authentic and ill-defined tasks are especially important characteristics of problem-based, case-based, and project-based learning environments (Albanese & Mitchell, 1993; Barrows, 1998). Since the students’ perceptions of the helpfulness of a general science learning environment was investigated in this study, students might not have experienced these types of tasks, and might have perceived them, therefore, as unhelpful in prompting their reflection.

Reflective Learning Environment as the Most Helpful Factor

Significant differences were found between the three perceived factors that prompted reflective thinking. The most helpful factor was the reflective learning environment, defined by these results as the various elements that make learning more active and student-centered. This is consistent with the design principles proposed in the constructivist learning environment literature (Jonassen & Land, 2000). Students’ perceptions of the reflective learning environment as the most helpful factor suggests that the design of educational activities intended to prompt reflective thinking should take into account elements that foster tolerant, active, collaborative, and experiential learning environments—those that build in learner control, and involve complex learning activities and working with partners.

Learner Control and Collaborative Learning as the Most Helpful Elements

Also of importance was that these students perceived the freedom to explore and work with partners as the most helpful instructional design elements. These two elements are referred to in the literature as learner control and the social nature of learning. The fact that students in this study perceived learner control as helpful is consistent with the results of previous studies that equated learner control with increases in instructional effectiveness and efficiency, learner independence and mental effort (Steinberg, 1989). These findings imply that learners think that they will have more opportunity to be engaged in reflective thinking if they are in control of their learning activities (Duffy & Cunningham, 1996; Williams, 1996).

As young teenagers develop into late adolescence, their social interaction with peers becomes a more important determinant of their behaviour and development of thinking skills (Craig, 1983). Previous research on collaborative learning has determined that active exchanges of ideas within collaborative learning groups promote the development of thinking skills (White, Shimoda, & Frederiksen, 1999). Small-group learning enables students to participate in reciprocal understanding. While participating in collaborative learning activities, students go through a process of conflict resolution, explanation, and justification before arriving at a shared conclusion (MacKnight, 2000). Therefore, reflective-thinking skills may evolve from social learning.

Furthermore, Johnson and Johnson (1996) suggest that collaboration and learner control are closely related to each other. They found that collaborative learning improves the effectiveness of learner-controlled environments. When students work with peers, they learn to control their learning situation, seeking elaborative feedback from each other and making effective instructional decisions, whereas when they work individually they do not. Thus, it is expected that the two design elements, collaboration and learner control, helped students to think that they would engage in reflective thinking. This may be the reason why these elements were perceived to be significantly different from all the other elements, but not from each other.

Direct Guidance Needed to Prompt Reflective Thinking

It is also interesting to note the low level of helpfulness ascribed to reflective writing (M = 2.94, SD = 1.13) despite the literature noting the importance of reflective-writing journals. Reflective writing and reflective question prompts were perceived as the least helpful of all the elements within the reflective scaffolding tools factor, but reflective question prompts were perceived as significantly more useful in prompting students to think reflectively than reflective writing.

The ranking of mean scores for the other derived factor, reflective teaching methods, also suggests learning needs for younger adolescents. The explanation-oriented method ( teacher explanations) was perceived as more helpful than the inquiry-oriented method ( teacher questions). These findings imply that young adolescents may feel a need for more direct guidance in reflecting on their learning.

Differences in Perceptions of Helpfulness by Grade Level

The data analysis revealed significance differences between the grades on two specific elements, concept mapping and reflective question prompts. Sixth grade students perceived concept mapping (drawing pictures ) as more helpful than eighth grade students did. These findings suggest the necessity of developmentally appropriate support for reflective thinking. According to the National Middle School Association (1995), young adolescents between the ages of 10 to 15 are in a transition period from concrete to abstract thinking. However, because cognitive growth occurs gradually, most students require ongoing concrete, experiential learning in order to develop intellectually. The finding that sixth graders perceived concept maps as a more helpful element of their learning environment than eighth graders did is consistent with this research; these younger adolescents are still at the concrete-thinking stage and can benefit from experiential learning. The result that seventh-grade groups perceived reflective question prompts as more helpful than the sixth-grade group, may show that as their intellectual abilities grow, students can become more involved in abstract thinking activities. These findings imply that we must design developmentally responsive environments that take into consideration the different characteristics of young adolescents. However, the fact that the authors used a design that was not balanced by grade level (6 th =85, 7 th= 40, and 8 th = 18), and did not use a random sample of participants, tends to limit the generalizability of this result. Since factor analysis solutions are sensitive to differences in the numbers of students in each grade level used in this study, it may be that the instructional methods used by teachers at the different grade levels influenced the student responses.

Differences in Perceptions of Helpfulness by Gender

The last finding of importance was the non-significant difference between boys and girls. Both genders perceived the most helpful factors and elements similarly. This result is partially supported by other studies on sex differences in the areas of learning and memory. According to Shepherd-Look (1982), there are no sex differences in a variety of learning processes, mainly in memory tasks. Although reflective thinking is one of the higher-order thinking skills, this finding suggests that likewise neither sex has superior higher order thinking skills in perceiving the helpfulness of reflective thinking.

Conclusion

This study identified three design factors that a sample of young adolescents perceived as helpful in prompting their reflective thinking. These factors were reflective learning environments, reflective teaching methods, and reflective scaffolding tools. The reflective teaching methods factor included in this study was perceived as significantly less helpful than the reflective learning environments factor, and the reflective scaffolding tools factor as significantly less helpful than the reflective teaching methods factor. Although the students responded to each of these factors individually, it is still unclear how the factors interact with each other. Further research is needed to understand the relationships among these three factors so that a theoretical model can be devised to explain the mechanism of the instructional environment as a variable in prompting reflective thinking. To develop such a sophisticated theoretical model that can be applied in various situations, further research would also be needed to examine students’ perceptions of the factors at different development stages (e.g., elementary-school children, college students) and within different learning environments (e.g., cases, projects, problem-based learning environments, etc.). Such a model could then be used to suggest instructional design factors that could be integrated into specific learning environments to prompt reflective thinking in students at a variety of age or grade levels.

One of the most meaningful findings of this study, however, was the perceived importance of learner control and social/collaborative learning in prompting reflective thinking. The students felt that being in control and working with partners helped them reflect more than the other activities did. Since the reflective learning environment factor emerged as the most helpful factor, it is important to further refine and test the attributes of this factor. This process will help to determine whether there are other specific components in the learning environment that students perceive as prompting their reflective thinking.

Another important finding from this study is that essential factors thought to prompt reflective thinking were validated with the factor analysis. Noting the scarcity of an instrument measuring students’ perception on reflection, the inventory developed in this study may provide other researchers with a useful tool to further study the construct empirically.

Although the middle-school students in this study perceived collaborative learning as the most helpful element in prompting their reflective thinking, they could still have difficulty in group decision-making and collaboration. In fact, some teachers who participated in this study reported that many of their students were inexperienced in group decision-making and collaborative learning. Often, teachers had to provide a great deal of scaffolding and coaching in such situations. Teachers also instituted problem-solving, decision-making, and team-building activities and experiences to help their students become more productive during collaborative sessions. This result implies that design elements of the reflective learning environment should be considered regarding delivery type or scaffolding type. Further studies should investigate how to scaffold reflective thinking during collaborative activities.

While there is more research to be done, the findings of this study have important implications for instructional design. Empirical evidence has been provided regarding students’ perceptions of the most helpful factors in prompting their reflective thinking. Student-centered learning environments that facilitate deeper thinking can be designed by including active, learner-controlled, collaborative, complex, and unhurried activities.

Acknowledgment

This project was made possible through funding from the National Aeronautics Space Administration, Leading Educators to Applications, Research, and NASA-Related Educational Resources in Science (LEARNERS), a Cooperative Agreement Notice from the NASA Education Division and Learning Technologies Project. Project Number: NCC5-432: Learning Using ERAST Aircraft for Understanding Remote Sensing, Atmospheric Sampling and Aircraft Technologies, (LUAU II).

References

Albanese, M., & Mitchell, S. (1993). Problem-based learning: A review of literature on its outcomes and implementation issues. Academic Medicine, 68(1), 68-81.

Aldred, M., & Aldred, S. (1998). Problem-based learning: The good, the bad, and the ugly, Journal of Dental Education 62(9), 650-655.

Andrusyszyn, M-A., & Daive, L. (1997). Facilitating reflection through interactive journal writing in an online graduate course: a qualitative study. Journal of Distance Education, 12(1), 103-126.

Atkins, S., & Murphy, K. (1993). Reflection: A review of literature. Journal of Advanced Nursing, 18, 1188-1192.

Boyd, E., & Fales, A. (1983). Reflective learning: Key to learning from experience. Journal of Humanistic Psychology, 23(2), 99-117.

Barrows, H. (1998). The Essentials of problem-based learning. Journal of Dental Education, 62(9), 630-633.

Butler , D., & Manning, M. (1998). Addressing gender differences in young adolescents. ERIC Document Reproduction Service No. ED 41560.

Craig, G. (1983). Human development. Englewood Cliffs, NJ: Prentice-Hall Inc.

Davis, E. (1998). Scaffolding students’ reflection for science learning. Unpublished doctoral dissertation. University of California, Berkeley.

Dewey, J. (1933). How we think. Boston, MA: DC Heath and Co.

Duffy, T., & Cunningham, D. (1996) Constructivism: Implications for the design and delivery of instruction. In D. Jonassen (Ed.), Handbook of Research for Educational Communications and Technology (pp. 170-198). New York, NY: Macmillan.

Gallagher, S. (1994). Middle school classroom predictors of science persistence. Journal of Research in Science Teaching, 31(7), 721-734.

Griffith, B., & Frieden, G. (2000). Facilitating reflective thinking. Counselor Education & Supervision, 40(2), 82-92.

Hair, J. F., Anderson, R. E., Tatham, R. L., & Black, W.C. (1995). Multivariate data analysis: With readings. Upper Saddle River, NJ: Prentice Hall.

Hmelo, D., & Ferrai, M. (1997). The problem-based learning tutorial: Cultivating higher order thinking skills. Journal for the Education of the Gifted, 20(4), 401-422.

Johnson, D., & Johnson R. (1996). Cooperation and use of technology. In D. Jonassen (Ed.), Handbook of Research for Educational Communications and Technology (pp.1017-1044). New York, NY: Macmillan.

Jonassen, D., & Land, S. (2000). Theoretical foundations of learning environments. Mahwah, NJ: Lawrence Erlbaum Associates.

Kinchin, I., & Hay, D. (2000). How qualitative approach to concept map analysis can be used to aid learning by illustrating patterns of conceptual development. Educational Research, 42(1), 45-57.

Klimczak, A., & Wedman, J. (1997). Instructional design project success factors: An empirical basis. Journal of Educational Research and Development,45(2), 75-83.

Koszalka, T. & Song, H. (2002). Examining learning environmental design issues for prompting reflective thinking in a web?enhanced PBL.A paper presented at AERA 2002 conference, New Orleans, LA.

Koszalka, T., Song, H., & Grabowski, B. (2002). Learner’s perception of design factors that support their reflective thinking during Problem-Based Learning. Proceedings of the AECT 2001 Conference, Research & Theory, Vol. 1, 217-221.

Lin., X., Hmelo, C., Kinzer, C., & Secules, T. (1999). Designing technology to support reflection. Educational Technology Research & Development, 47(3), 43-62.

MacKnight, C. (2000). Critical thinking and collaborative inquiry. Journal of Interactive Instructional Development, Spring, 3-11.

Malpine, L., & Weston, C. (2000). Reflection: Issues related to improving professor’s teaching and students learning, Instructional Science, 28(5), 363-385.

Moon, J. (1999). Reflection in learning & professional development: Theory and practice. London: Kogan Page.

National Middle School Association. (1995). This we believe: Developmentally responsive middle level schools. Columbus, OH: National Middle School Association.

Nunally, J.C. (1978). Psychometric theory (2 nd ed.). New York: McGraw-Hill.

Rowe, M. (1974). Wait-time and rewards as instructional variables. Journal of Research in Science Teaching, 11, 81-94.

Roeser, R. W., Eccles, J. S., & Sameroff, A. J. (2000). School as a context of early adolescents’ academic and social-emotional development: A summary of the research findings. The Elementary School Journal, 100(5), 443-471.

Schön, D. A. (1991). The Reflective practitioner. San Francisco, CA: Jossey-Bass.

Shepherd-Look, D. L. (1982). Sex differentiation and the development of sex roles. In B. B. Wolman (Ed.), Handbook of Developmental Psychology (pp. 403-432). Englewood Cliffs, New Jersey: Prentice-Hall.

Steinberg, E. (1989). Cognition and learner control: A literature review, 1977-1988. Journal of Computer Based Instruction, 16(4), 117-124.

Stepien, W., & Pyke, S. (1997). Designing problem based learning units. Journal for the Education of the Gifted 20(4): 380-400.

Tabachnick, B., & Fidell, L. (2001). Using multivariate statistics (4 th Ed.). Boston: Allyn and Bacon.

Wittrock, M. (1990). Students’ thought processes. In M. C. Wittrock (Ed.), Research in Teaching and Learning, Vol. 3 (pp. 4-42). New York: Macmillan.

Williams, M. (1996). Learner control and instructional technologies. In D. Jonassen (Ed.), Handbook of Research for Educational Communications and Technology (pp. 957-983). New York, NY: Macmillan.

White, B., Shimoda, T., & Fredericksen, J. (1999). Enabling students to construct theories of collaborative inquiry and reflective learning: Computer support for metacognitive development. International Journal of Artificial Intelligence in Education, 10, 151-82.

Wolman, B. (1982). Handbook of developmental psychology. Englewood Cliffs, New Jersey: Prentice-Hall.

Virtanen, P., Kosunen, E., Holmberg-Marttila, D., & Virjo, I. (1999). What happened in a PBL tutorial session? Analysis of medical learner’s written accounts. Medical Teacher, 21(3), 270-276.