AJET 18(2) Lyons and Milton (2002) - learning to evaluate - evaluating to learn

Australian Journal of Educational Technology
2002, 18(2), 187-207.AJET 18
Learning to evaluate - evaluating to learn
Judith Lyons and John Milton
RMIT University This paper reports on the planning and implementation of an evaluation of the final prototype of a physical and computer based simulation in a postgraduate midwifery program. The evaluation framework was designed to reflect the pedagogy of Laurillard‘s conversational framework, which had been used to structure the learning experiences in the simulation. Data collection methods and analysis highlighted the themes of discussion, interaction, reflection and adaptation of student learning actions as well as intrinsic feedback - all central to the conversational framework.
An action learning approach to evaluation planning and implementation involving an external mentor is described in the following paper including the relationship to the conversational framework, the findings on the student learning environment and the student learning processes and outcomes. The lessons learned about evaluation are also elaborated.
Introduction
This paper reports an evaluation of a prototype teaching development initially funded by the Committee of University Teaching and Staff Development (CUTSD) for a postgraduate midwifery program at RMIT. The Nurses Board of Victoria also supported the evaluation during design and development phases of the project.
The Pregnancy Simulator Learning Package (PSLP) is a complex interactive, multimedia computer and physical simulation designed to help teach midwifery students pregnancy assessment skills. The assessment of the pregnant woman is one of the crucial skills for health care professionals. Midwifery practitioners use abdominal assessment to identify progress of pregnancy and labour, determine the size and position of the fetus, and to listen to fetal heart rate. The findings give the midwife a good indication of fetal well being, and how pregnancy and labour are progressing. Inaccurate assessment may result in poor outcomes for mother or baby.
In what follows, the background presents the broad educational and professional issues that led to the development of the PSLP as well as the educational context in which PSLP is integrated into the midwifery program. This is then followed by a description of the components of PSLP as a technology based learning environment with an emphasis on the teaching-learning framework of PSLP. Finally the evaluation framework and PSLP evaluation findings are presented. Background
Educational and professional issues
Various issues encountered in conventional teaching in university midwifery programs were the impetus for developing PSLP. Learning to be a competent nurse or midwife is based on engaging in dynamic experiential learning with actual patients and one that captures the reality of professional practice. However, clinical learning contexts introduce hurdles and barriers for the student and the clinical teacher due to shortage of staff and problems with practice placements. Hospital staff are less committed to students‘ learning because having students places extra burdens on them (Castledine, 2001). Furthermore, clinical teachers may not be adequately prepared to teach in clinical settings (Scanlan, 2001; Greslish, 2000). Also, supervised clinical practice experience can increase student anxiety, especially if the women express pain and discomfort during the assessment, or fail to give consent for a teaching session using them. Ethical considerations, privacy issues and time constraints can reduce the extent and quality of immediate feedback. Performance feedback is often given in another room at a much later time, thus reducing learning opportunities for adaptive practice experience. (Lyons, Miller & Milton, 1998).
The educational context
The transfer of midwifery education to the higher education sector from hospital based programs was completed in 1994. The challenges to date have been to provide a cohesive educational experience that provides the tenets of higher education with its development of broader graduate capabilities, and acquisition of professional competencies that requires ‘hands on‘ clinical experiences, all within the limitations of a one year program.
The Graduate Diploma in Midwifery program in Australia is mainly offered as a one year postgraduate nursing degree. Midwifery students entering the program must be registered with the State Regulatory Authority (Nurses Board of Victoria) and have at least a year‘s work experience in acute care nursing. Although there are major benefits overall in having experienced nurses and midwifery as a postgraduate program, it also creates issues for students‘ academic workload when delivering new midwifery discipline knowledge in a limited time frame, as opposed to running a three year undergraduate midwifery program as it is offered in some Australian states and internationally.
University midwifery programs generally consist of a one-third theoretical component delivered over two days and two-thirds clinical component which is delivered over two days, within academic semesters and a six weeks block supervised clinical practice during semester breaks.
The pregnancy and health assessment is a major curriculum focus covering two midwifery specific courses, namely, antenatal, and labour and birth aspects of professional practice. The curriculum focus of pregnancy and health assessment is holistically presented as a women centred continuity of care model, thus retaining continuity between childbirth in normal and complicated pregnancy, labour and birth, and postnatal childbirth experiences. The principles and concepts of abdominal assessment are generally presented to students by way of lectures, tutorials and/ or demonstrations with practice on static models in laboratories or on human volunteers in the hospitals and clinical agencies. Learning occurs at varying rates using these traditional strategies, and showing students a model representation of a baby in the abdomen has been found useful in facilitating learning. However, the poor quality of static plastic models does not provide appropriate tactile simulation for building the analytical and interpretive skills associated with physical examination. Students often identify the psychomotor steps in performing abdominal palpation but experience difficulty in interpreting assessment data, recognising the relationships and in distinguishing its significance. They fail to relate the physical touch of palpation in assessment, the noticing of crucial clues, the significance of the findings and their impact on the midwifery care of the woman and her baby. Anecdotal evidence and competency based assessments demonstrated that students lacked confidence in their ability to perform pregnancy assessment and abdominal palpations, often requesting added reassurance and supervision.
The PSLP complements current teaching models and is integrated in two courses over two semesters and is undertaken in parallel with lectures, tutorials, and clinical practice. The three hours every week scheduled for each course has two hours of lecturers followed by an hour of self directed learning including group work, class presentation preparations, and computer learning activities that include PSLP. Students self roster into small groups to use the PSLP simulation. However, students can also use it more often or individually prior to a specific clinical rotation or competency assessments if they so desire.
In second semester the PSLP is embedded in an intensive set of learning activities in all courses, including extensive use of online BlackBoard course resources, online library resources, and online synchronous and asynchronous discussions on a pregnancy related Internet site, electronic journal article critiques, and clinical and professional issues. Students also use email facilities and a major online role play on WebCT. Previous experience with learners in this program has revealed that they require time to adapt to using computer technology. There are issues of fear, trepidation, and lack of basic computer skills to be addressed.
Online learning, including PSLP use, contributes ten percent of the total grade. The marks are based on students‘ hypothesis and rationale for their health assessment (as well their computer skills), rather than on the correct answers to the simulation. The Pregnancy Simulator Learning Package
The PSLP includes a computer controlled model of a pregnant abdomen at term that is linked to computer generated, interactive learning activity cycles (see Figure 1 below). The physical model consists of a term fetus that rotates into various positions simulating fetal accommodation in the womb, and students palpate the abdomen to obtain information about the baby and pregnancy.

Figure 1: Pregnancy Simulator Learning Package (PSLP)
The multimedia tutorial simulation presents case scenarios to assist students to identify and address misinterpretations, and develop a comprehensive understanding of assessment concepts and clinical decision making skills. Students in small groups work through these case scenarios. There are four case studies that can be selected from the main menu, and upon selection, the learning package controls the physical model to set the specific requirements of the case study.
The first simulation is of is Mrs Grant who represents a normal pregnancy and is designed to teach novice students the concepts of pregnancy, health and abdominal assessment. The learning experience and the feedback provided in this case study guide the students in a predetermined sequence to enable them to interrogate the woman‘s case history, inspect and physically palpate the model. The other three case studies are hierarchical in nature and increase in complexity to present variations in normal pregnancy assessment findings, abnormal pregnancy findings and medical and obstetrical conditions that may complicate the pregnancy.

Figure 2: Conversational framework (simplified from Laurillard, 1993) The teaching-learning framework of the PSLP
Teaching strategies based upon discovery and problem solving are an integral part of the learning package (Milton and Lyons, 1997). The learning experience is designed as a dynamic combination of the experience of the physical act of palpation (psychomotor skills), and the interpretive process of conceptualising, visualising and clinical performing decision making (cognitive).
Essentially the teaching-learning process on which PSLP is built is a dialogue between the teacher and the student and a student to student dialogue (Milton and Lyons, 1997). The model for the former is explicitly designed on the basis of Diana Laurillard‘s conversational framework (1993). It reflects a desire that the dialogue be discursive, adaptive, interactive, and reflective (Figure 2). A key goal of the package is for students to receive intrinsic feedback as part of their interaction with the simulation: the students come to see that their way of thinking is inadequate in explaining the behaviour of simulated pregnancy.
We have described elsewhere how the PSLP realises the demands of the conversational framework (Milton and Lyons, 1997). A brief description below maps a PSLP learning sequence to Laurillard‘s framework (see Figure 3). These sequences are repeated for each pregnancy assessment concept to be learnt, and are presented with a combination of video, still images, text and sound. The teacher (PSLP) presents the overall conception which is broken down into smaller, manageable learning tasks or concepts, enabling learners to outline their goals for conducting each aspect of comprehensive health and pregnancy assessment. Short open answer questions elicit learners‘ conceptions and explanations in their own words, as they interrogate information in the case notes to predict what they may discover on physical examination, and provide a rationale for their hypothesis. Following this, students perform the palpation and input their findings into the computer. Learners select the teacher‘s redescription of concepts through multiple choice questions, which offer the teacher‘s conceptions and enable the program to recognise students‘ responses and explanations. The multiple choice questions, with the teacher‘s knowledge built into the underlying database, enacted the teacher‘s feedback both at the discursive and interactive levels. This redescription of concepts also encourages learning of professional language for communication in healthcare practice.

Figure 3: PSLP Learning sequence flowchart
The focussed feedback presented through text, video and audio is the crucial element of the PSLP learning task, because it enables students to reflect on their actions, repeat actions and to modify them appropriately. Feedback is not simply a correct or incorrect response, but a combination of hints, clues and expert knowledge constructed using previous students‘ common misconceptions to suggest things to try, or interpretations to contemplate. Feedback can also draw on additional resources for students to interrogate, to enhance their learning, for example case notes, or a video of how to place your hands when conducting a specific palpation technique.
Feedback guides the students through critical learning pathways based on their responses and is designed to enable students to discover concepts and solve problems in order to complete the learning activity prior to moving onto the next learning sequence. Students‘ responses and choices are saved as student notes to provide a means for teachers to reflect on students‘ performances and add further focussed feedback pathways immediately after the learning sessions if necessary. Further description of PSLP is provided in Lyons, Miller and Milton (1998) and Lyons and Milton (1999). Constructing an evaluation framework
Although PSLP was designed with the conversational framework in mind, we did not intend to evaluate the framework solely but approached it as a coherent framework within which to evaluate both the pedagogical benefits (learning outcomes) and the ways in which students used the new tool to facilitate their learning (learning process). Our evaluation focuses on student use of the learning package - how the conversation actually unfolds in using the PSLP and how this conversation is associated with learning outcomes. We approached the evaluation intent on uncovering student engagement in the key conceptions to be learned and to uncover what the student is attending to, and the strategy they employ in learning, at any point in time. It is with this frame of reference that we sought to understand the crucial processes of learning. Lesser importance was placed on evaluation of the learning environment created by PSLP - testing its features, functionality and usability - because these were examined extensively during the design and development phases of the PSLP project.
Although we could readily access student actions (for example, by observation), this evaluation paradigm clearly pointed to post-learning interviews with students - to ascertain their intentions as well as descriptions of their actions. Equally important, the interviews could ascertain the students‘ interpretations of the teacher controlled dialogue represented in the PSLP. In this manner we could relate learning outcomes back to approaches to learning and then further back to interpretations of the teacher dialogue. Only in this way could we identify areas and ideas for improvement. It clearly extends beyond a narrow focus on process. It also potentially opens the evaluation to the impact of one student on another - not an explicit part of the conversational framework. Preparing and carrying out the evaluation plans
Evaluation plans for each semester were prepared, discussed and agreed between teacher and mentor. The plans listed each component of evaluation, their aims and the principal items of qualitative evaluative data being sought. The evaluation instruments and regime for application were determined prior to the commencement of student learning with the PSLP.
The evaluation was conducted in its own reflection cycle leading to improvements to the PSLP and to some changes in evaluation methods and tools during the two semesters. So, for example: the observation checklist proved to be of no use (as explained below); the interviews quickly came to probe specific points of the students‘ experiences. The observation helped us to identify which parts of the experience to concentrate upon in the interview; the video was found to be of limited value given the close observation by the teacher followed by the immediate interview.
Analysis proceeded in parallel with the conduct of the evaluation. Regular meetings of the discipline expert/ teacher and the evaluation mentor were vital to make sense of the emerging data and to decide appropriate improvements to the PSLP.
On all aspects of evaluation of PSLP there was mutual collaboration and continual critical dialogue between the discipline expert/ teacher and the evaluation ‘mentor‘ (apart from other things, this allowed for continual reflection on the learning theory underpinning the learning experience as an integral part of the analysis). Evaluation questions
We constructed our evaluation questions using the principal themes of the conversational framework - discursive, adaptable, interactive and reflective - rather than the stages of conversational model outlined in the Figure 3 flowchart earlier. The teaching and learning process questions are holistic in the sense that they explore all relationships within the conversational framework. They permitted us to work comfortably with students‘ interpretations as well as creatively explore ideas for improvement. Four key questions, together with a subset of guiding questions reflecting the key questions, were used to collect qualitative data.
The first question centres on the students‘ conceptual knowledge and includes student responses to feedback provided in the form of the (teacher‘s) re-descriptions of student conceptions. This question encompasses all student led interactions and interpretations involving student conceptual knowledge: the top right hand side of Figure 2. The second question relates to the students‘ experiential knowledge: the bottom right hand side of the conversational framework. Most important, it includes student responses to feedback provided by and within the simulation (intrinsic feedback). These two questions required data about students‘ experiences, actions, descriptions and interpretations. The third question on learning process covers the totality of the package including, but not limited to, the teacher‘s own reflection and adaptation built into parts of the package (left hand side of Figure 2). The final question evaluates learning outcomes. The four key aspects included: With what intentions and how did the students engage with the theoretical concepts and knowledge? This naturally involved reflection and subsequent adaptation of actions.
With what intentions and how did the students interact with the teacher constructed world of the simulation? Again reflection and subsequent adaptation of actions is an aspect of this question.
How do the teacher‘s task goals, feedback and adaptations keep the student engaging with the key conceptions, reflecting on their interactions and being constructively challenged in the world of the simulation?
What knowledge and skills did the students have when they entered the course and what had they learned upon conclusion of their use of PSLP?
Data collection and analysis
Twenty two students in groups of two or three participated voluntarily in both semesters. In first semester, participants concentrated on one learning cycle addressing the lie of the fetus, a concept associated with antenatal care. The evaluation in second semester examined all palpation cycles and concepts significant to pregnancy, labour and birth.
Data collection included a variety of sources namely pre and post tests, participant observation and observer notes, video recording, audiotape of participant interviews, student notes and responses in PSLP and course evaluation. The pre- and post-tests concentrated on students‘ knowledge of pregnancy and health assessment concepts. The participants were observed and observer notes made on all occasions that PSLP was used in first and second semester. The checklist developed to assist observation was used only in first semester. Observation was conducted from an office away from where the PSLP was located. Video recording of all participant groups using PSLP was conducted in first semester only. Audio taped post-PSLP interviews were conducted with pairs and groups in another office immediately after each learning session. Semi-structured questions and sub-questions were used during interviews, informed by data from the PSLP interactions and observer notes.
Table 1: Basic evaluation framework
Evaluation questionEvaluation data source
Q1. Student engagement with conceptionsStudent responses to open answer questions recorded on the student data file within the PSLP. Student selection of multiple choice options both observed and recorded on the data file. Observation of dialogue between student pair. Audio-video recording of student dialogue while each pair uses the PSLP. Observation by discipline teacher expert.
Q2. Interaction with simulated world including intrinsic feedbackObservation by discipline teacher expert. Audio-video recording of student interaction. Student pair interview immediately following use of the PSLP.
Q3. Teacher adaptation of the world and reflection on performanceObservation of response of PSLP in relation to students‘ descriptions and actions. Teacher diary throughout evaluation*
Q4. Students‘ prior knowledge (and skills as self reported)Written response test conducted prior to use of the PSLP Pre PSLP questionnaire on student background
Q4. Student learning outcomesPost PSLP test Course assessment. Self report - post interview
* An important finding of our approach to evaluation has been that teacher adaptation and reflection were in fact partly addressed by the evaluation itself. We take up this point later in this paper.
The interview data were transcribed, coded and analysed by the evaluators using NUD*IST. The coding involved the most frequently occurring themes, which were later collated and grouped together in relation to learning process and learning outcome key questions. The themes were also coded in relation to the discursive, adaptable, interactive and reflective themes of the conversational framework. The observer notes and student interactions in the PSLP were also classified according to themes and adaptations to responses made by the students. A list of the evaluation methods related to these questions is presented in Table 1.
In addition: the teacher‘s diary, as well as regular interaction with the evaluation mentor, provided a crucial source of creative ideas as well as forming an integral part of the analysis.
evaluation of the whole course conducted at the end of the semester included focus group discussions and individual feedback.
Evaluation outcomes: What have we learned about student learning processes and outcomes?
Emergent themes
The evaluation demonstrated that students engaged with the learning task through their use of the PSLP and in collaboration with each other on the learning task. Each of the four themes of Laurillard‘s conversational framework was evident. The most common themes were: Discussion (with sub-themes of discuss, debate and understanding the learning task to set goals). Interaction (with sub themes of action, interrogating information, drawing conclusions, engaging with selection options, reviewing actions, writing in free text and palpation findings); Reflection, including drawing comparisons on clinical practice, class lectures and pace of learning; Adaptive learning (with sub themes of going back to review, palpate or change responses); Feedback (with sub themes of getting used to a different type of feedback, wanting to know how many correct or incorrect responses, and their actions following feedback); Collaboration, (with sub-themes of working in groups, discussing, debating, checking answers and gaining consensus). Computer experiences, including comments on level of computer skills, expectations of learning with computer and in non clinical setting, features of interaction with the PSLP (say, write and do the right thing; competing with the computer - ‘the truth is out there‘); PSLP learning environment, which included the effect of questions and feedback in the package on their actions; the pace of learning; the ease or difficulty of use; functionality, wording of instructions and usefulness. Learning outcomes (with sub-themes of think and rethink, making us describe, achieving clearer understanding, concentrate on, knowledgeable, consolidate learning, increased confidence and assessment skills); Finally there were positive and negative emotional responses to the PSLP learning experiences.
Aspects of a surface approach to learning were evident as some students struggled with the discovery approach to learning inherent in the conversational model. However, most students demonstrated a deep approach to learning in which they interacted with the learning task, collaborated, discussed, modified their actions, and reflected on their learning and compared it to other learning experiences such as clinical practice. Student engagement with conceptions
Before interaction with the PSLP, many students were either ignorant of or expressed misunderstandings about key concepts such as fetal lie, fetal attitude and fetal position. Although students knew how to palpate and what techniques to use, they had difficulty in recognising and interpreting what they were feeling. For example, fetal attitude is the relationship of fetal head and limbs to fetal trunk and must be determined first to differentiate the various fetal cephalic presentations. With a baby‘s head in the mother‘s pelvis, the degree of flexion of the head will determine what aspect of the head is leading into the pelvis, hence whether it is a cephalic vertex (flexed attitude with chin on chest), brow (incompletely deflexed attitude), or face presentation (deflexed attitude) and these alternatives lead to different pregnancy and birth outcomes. One student commented: "You know what you are doing like when you are feeling [palpating] but to link that to what the concept is - is difficult. Like the description thing. You don‘t think in terms of the concept when feeling the ‘attitude‘ [a midwifery term]. I know now you are feeling the head not the attitude which tells you the ‘presentation‘ [a midwifery term] but this forces you to think of it and link to theory, not just do it." Another example of misunderstanding was in determining the fetal position which is the relationship of the denominator of the presenting part of the fetus to the maternal pelvis. Students often indicated fetal position of cephalic presentations from fetal back and not from the denominator on the head which is in the pelvis and not the back. Some students also failed to understand that fetal position cannot be identified if the head has not entered the pelvis because there is no relationship to determine that position. Feedback made this self evident to students, enabling them to integrate theory into practice and modify their answers appropriately on repeat attempts.
Clear evidence of effective learning is to hear students relating academic knowledge and descriptions to practice and vice versa. Academic knowledge is then likely to be ‘internalised‘, becoming central to the student‘s experience as a midwife: "I find in practice you know what you are doing and you can work it out but it is difficult to translate what you find into words and describe it because you do not understand what you are talking about and this computer thing makes you think of how to describe it" Interaction with the PSLP simulated world including intrinsic feedback
Interaction in the PSLP package required students to set goals, predict findings based on current information, interrogate case notes, inspect the abdomen and palpate to discover the accurate information related to a specific case study. Students‘ interactions were observed and well documented in their notes. However, some students commented that writing free text responses was a difficult task because it required considered thought and organisation to provide appropriate responses, and they therefore preferred selection of options in multiple choice questions. It was also observed that on repeat attempts, students often bypassed text input and paid more attention to the selection options, debating each option and relating it to what they had felt on the model during palpation.
The intrinsic feedback incorporated into the PSLP required students to explore the world of the simulation and case study presented by the computer. Feedback through suggestions, cues, added expert knowledge, and interpretations to try, were a new way of engaging with the key ideas for most students. For example, prompting students that it was impossible for fetal back and limbs to simultaneously accommodate the same side of the abdomen and that the fetal lie was the relationship of the baby‘s back to the mother‘s uterus, made students go back and palpate again, attending to the location of fetal back, to modify their answers appropriately. Initially students found it frustrating and wanted to know immediately how many of their selections were correct or incorrect. However, having adjusted to this mode of feedback it was observed that they were paying particular attention to feedback hints and clues, intrinsically acknowledging when their response was wrong and required them to review their actions. This was also evidenced through the decreased repeat attempts of learning sequences, quicker progression through the learning activity, and fewer remarks during interviews in second semester about the desire to know if their responses were right or wrong. Significantly, students recognised that it was not straightforward correct or incorrect feedback: "...makes you question yourself and you go back and check what you did again and change your answers"
"It makes you think all the time - when it (computer) does not like our answer it makes you think that something is not quite right about your palpation and makes you think of what you have to do."
"We had to go back and review our work. Feedback makes you think harder"
Furthermore, an interesting observation was that some students critically engaged with the computer by competing with it, trying to give ‘perfect‘ answers, insinuating that the ‘true‘ answer was in the computer, and second guessing the answer that the computer might provide.
The midwifery students have extensive experience as nurses in clinical practice and are engaged in a formal work experience initiative as part of the program. We found students linking their learning to their domain of practice. The distinction between the academic world of the PSLP and the world in the clinics was being broken down: "What you are most interested in and what you are looking for. It makes you think back on your clinical practice, and why you are doing what you have been doing, and what you can do better next time you go out [to clinical practice]."
"It was helpful doing the computer package. I would rather have done the computer package before I go out to clinicals. I would have the knowledge and would know what to expect when I meet the woman there."
Teacher adaptation of the world and reflection on performance
Analysis of post-PSLP interview data and student learning notes provided opportunities to reflect on students‘ learning. In addition, the teacher‘s notes on the observations of students‘ interaction and engagement with the learning task enabled the teacher to reflect on students‘ actions and descriptions, and to adjust the PSLP learning environment to make it more meaningful to students.
Although the physical simulation (teacher constructed world) was not criticised, three aspects linked to the world of action were raised as concerns and were adjusted for the students. Firstly, we improved the default feedback for occasions when students may have drawn a correct conclusion but made an error in their rationale. Secondly, we ensured that feedback based on common mistakes was presented before feedback about less common errors. Finally, we modified some questions and tasks to ensure that students could understand what was being asked of them. Student reaction in second semester indicated that the earlier concerns had been alleviated. Student prior and post knowledge and skills
Students‘ midwifery knowledge and skills improved markedly from pre- to post-test, but this improvement cannot be totally attributed to the PSLP alone, because of the integrated nature of the learning experiences within the program. Nevertheless, comments showed that students learnt the conceptual and psychomotor skills targeted by the conversational framework of PSLP: "...Would have learnt assessment skills not just physical ones but mental ones. So that when they actually went out [to clinical experience] they would have in their mind ‘this is what I have to do; this is what I know; I have to do these things and these are things that can be different from the norm" Collaborative learning
We must comment on collaborative learning, which is not incorporated into Laurillard‘s conversational framework. A recurring theme was the importance of students learning together. Students claimed, "... We really discussed and learnt from each other." Discussion as a learning strategy was used in all aspects of the learning cycle, including during their decision making, inputting their results and following the feedback. "Somebody thinks they are right and the other one thinks that they are wrong then you have to discuss. - You have to make sure. That is what I like about group work. They can say you are completely wrong or completely right and you have to put in something if you get it wrong then you go through the discussion process again".
"We discussed the feedback" - "If the computer was telling me that I was doing it wrong I will go back and do it again... I might say the head is flexed and you [addressing fellow student] might say it is ‘military‘ we discuss it before we put it down and if it is wrong the computer will tell us but at least we discussed it."
What have we learned about evaluation methods?
The following issues and conclusions are based on the progressive evaluation of the PSLP during both semesters.
We found that an observation checklist was of no value. If your focus is the relationship between interpretation and action of individual students, observation needs to be undertaken by a person with discipline knowledge. That observer needs to be looking for student decisions which are leading them down inappropriate and unproductive paths. These are points which need to be followed by close questioning (eg ‘Why did you choose X? What were you thinking at that time? What was that question X asking you?).
When conducting formative evaluation on prototype software as part of a learning program, it is very important to: use the learning experience as supplementary rather than as the primary learning experiences;
fully inform students of the nature and purpose of the evaluation and, more particularly, of the role of the learning experience in their learning in the course.
We recommend this matter be raised for open discussion with students several times during the course.
If one‘s interest is student misconceptions arising from the use of the learning experience, it is important that a person familiar with the discipline (if not the discipline expert themselves) undertakes the interviews. Only that person can quickly and easily follow student descriptions of what they were attending to and, significantly, relate students‘ answers to possible misconceptions, which then influence follow up probing questions. The one reservation is that the interviewer has to ‘bracket‘ their own preconceptions to fully hear the students‘ voices and ‘read‘ their experiences.
It was mentioned earlier that the PSLP was just one part of technology based learning experiences in the second semester course. This raised issues for students and influenced their reflections on the PSLP at semester end. Accordingly, we argue that the teaching team should conduct an evaluation of the whole of the course, even if their primary interest lies in one of a few major learning experiences. This is an important part of summative evaluation and has a considerable bearing on ongoing implementation. So, for example, it would help the team decide how the learning experience should be presented to students and its relationship to other elements of the course.
With multiple data sources it is important to keep all evaluation data clearly labelled in a structured manner. In an evolving and ongoing evaluation, dates must be clearly labelled on all documents. In collecting the data, academic teachers have to initially ‘bracket‘ prior knowledge and understanding of students and their conceptions and misconceptions of the topics. The educational designer, divorced from everyday action with the students, has a ‘distance‘ from which new ideas and questions can be posed. In our case it helped keep the conversational framework firmly in mind at all times. Contrary to the view of some teachers that students are vulnerable and say what their lecturers expect, we found students were forthcoming and reported both negative and positive experiences.
Analysis, as much as the design of the evaluation itself, is a holistic process. That is, while we can focus on matters of detail (eg ‘how is this student engaging with the learning task, interpreting and acting in relation to that computer learning task instruction?‘), we should keep coming back to the overall perspective on teaching and learning, as well as the aims of the evaluation (eg ‘in acting or responding that way, how is the student seeing her relationship to the woman and her care?‘). What have we learned as teaching professionals?
Our evaluation had an interesting twist to it due to the fact that we were sharply focused on the dialogue between student and teacher. In a real sense we were aiming to replicate the complexity of a one to one, face to face dialogue. Our intention at most times was to adjust the instructions and feedback the ‘computer‘ gives the students, according to their actions and interpretations. Although the teacher‘s reflection and ‘fine tuning‘ of statements, questions, directions and feedback to students were conducted after the initial learning was completed and hence were pre-emptive in character, they nevertheless emphasised the importance of evaluation in teaching and learning scholarship (refer to the left hand side of Figure 4). Such evaluation also highlights the teacher‘s responsibilities in the teaching-learning process to ensure that the teacher‘s conceptual knowledge is made explicit to students so that students can detect and correct their misconceptions.

Figure 4: Evaluation related to Diana Laurillard‘s conversational framework
We found an evaluation paradigm embedded in an enhanced action inquiry approach fostered mutual collaboration and continual critical dialogue between the mentor (educational designer) and the mentee (discipline expert). The action inquiry approach complemented the conversational framework. We reflected critically on the key ideas that needed to be communicated to students, and spent time discussing our own paradigms, clarifying our own positions and explicitly examining the assumptions underlying our evaluation approach. According to Holter and Schwartz-Barcott (1993) and Sturt (1999), an enhancement action inquiry approach begins by working in a mutually collaborative way, but takes the process further to engage in critical dialogue to raise the collective consciousness about the phenomenon - in this instance the evaluation of computer facilitated learning. This meant the project took more the form of a partnership between an educational designer and discipline expert-academic teacher.
Our interest was with qualitative descriptions of students‘ descriptions, interpretations and actions. This qualitative approach to evaluation conducted in this naturalistic way also provides data with richness and better understanding of student learning facilitated by computers (Alexander and Hedberg, 1994; Bain, 1999; Guba & Lincoln, 1989; Ehrmann, 1990; Britain & Liber, 1999; Patton, 1990).
This approach also placed evaluation as an integral part of effective teaching and learning scholarship and prompted the teacher to critically reflect on her practice to improve student learning experiences and outcomes (refer to the left hand side of Figure 4). It emphasised evaluation as a continual process situated in the total learning experience of the students. Thus, learning centred evaluation was not just evaluation of the educational media but of the learning environment and the student learning process. Conclusion
It is crucial to select the evaluation approach, methods and analysis explicitly based on the perspective on learning and teaching that is informing the design and development of the learning experience. If the underlying view of teaching and learning is Diana Laurillard‘s conversational framework and its related assumptions, they will help you design the evaluation and should be a key reference point in your analysis. The whole integrated process of design, development and evaluation becomes an absorbing and fulfilling experience of learning in more ways than one. Acknowledgements
We would like to acknowledge the perceptive comments of the reviewers of the first draft of this paper assigned as part of the ASCILITE Staff Development Evaluation Project: Tony Gilding and Catherine McLoughlin. References
Alexander, S. & Hedberg, J. G. (1994). Evaluating Technology-based learning: which model? In K. Beattie, C. McNaught & S. Wills (Eds), Multimedia in Higher Education: Designing for Change in Teaching and Learning. Amsterdam.
Bain, J. D. (1999). Introduction: Learning-centred evaluation of innovation in higher education. Higher Education Research and Development, 18(2), 165-172.
Britain, S. and Liber, O. (1999). A framework for pedagogical evaluation of virtual learning environments. JTAP JISC Technology Applications Report. University of Wales: Bangor.http://www.jisc.ac.uk/jtap/htm/jtap-041.html [verified 24 Jul 2002]
Castledine, G. (2001). Clinical teaching is the key to improved practice. British Journal of Nursing, 25(7), 1307-1308.
Ehrmann, S. (1999). Flashlight Evaluation Handbook. Flashlight Project. http://www.ctl.wsu.edu/CTLSilhouette/mode/author/flashlight/EvaluationHandbook [viewed 24 May 2000, URL not found 25 Jul 2002]
Grealish, L. (2000). The skills of coach are an essential element in clinical learning. Journal of Nursing Education, 39(5), 231-235.
Guba, E. G. & Lincoln, Y. S. (1988). Do inquiry paradigms imply inquiry methodologies? In D. M. Fetterman (Ed), Qualitative approaches to evaluation in education: The silent revolution (pp89-115) New York: Praeger.
Holter, I. M. & Schwartz-Barcott, D. (1993). Action research: What is it? How it has been used and how it can be used in nursing? Journal of Advanced Nursing, 18, 298-304.
Laurillard, D. (1993). Rethinking University Teaching. London: Routledge.
Lyons, J., Miller, M. & Milton, J. (1998). Learning with technology: Use of case-based physical and computer simulations in professional education. Contemporary Nurse, 7(1), 98-102.
Lyons, J. & Milton, J. (1999). Recognising through feeling: A physical and computer simulation based on educational theory. Computers in Nursing, 17, 114-119.
Milton, J. & Lyons, J. (1997). Designing a physical and computer-based simulation based on educational theory. HERDSA Conference, Adelaide.
Patton, M. Q. (1990). Qualitative Evaluation and Research Methods (2nd ed) Newbury Park, CA:Sage.
Scanlan, J. M. (2001). Learning clinical teaching: Is it magic? Nursing and Health Care Perspectives, 22(5), 240-251
Sturt, J. (1999). Placing empowerment research within an action research typology: Methodological issues in nursing research. Journal of Advanced Nursing, 30(5), 1057-1063.
Authors: Judith Lyons, Midwifery Programs Coordinator, Department of Nursing and Midwifery, RMIT University
John Milton, Learning Technology Services, RMIT University
Please cite as: Lyons, J. and Milton, J. (2002). Learning to evaluate - evaluating to learn. Australian Journal of Educational Technology, 18(2), 187-207. http://www.ascilite.org.au/ajet/ajet18/lyons.html
[AJET 18 ] [AJET home ]
HTML Editor: Roger Atkinson [rjatkinson@bigpond.com]
This URL: http://www.ascilite.org.au/ajet/ajet18/lyons.html
Created 16 Aug 2002. Last revised 27 Aug 2002.