11.12. Evaluating Student Mastery and Program Effectiveness

Editorial

This month’s article by Maggie McVay Lynch is an adaptation of a chapter from her upcoming book The Online Educator: Creating the Virtual Classroom and spotlights the crucial role of faculty in the accurate assessment of student learning. Accreditation requirements have, for some time now, included the notion that integrated learning assessment plans, based on measurable learning outcomes, must be set forth at course, program, department, and institutional levels to reflect the strategic mission of the institution.

It doesn’t matter what the intentions of an institution’s upper administration may be; meeting these requirements ultimately rests with individual faculty and their desire, knowledge, and skill in creating learning outcomes and accurately assessing student mastery. It is only they who can reconcile the disparities that often exist among what the student is assumed to be learning, content delivered and practice assigned during a course of study, and make decisions about how the student is held accountable for their learning, when traditional tests and examinations may not be the best methods. This is a tall order for a group of experts most often hired for their content expertise and rarely for their formal preparation in the intricacies of teaching and learning.

Maggie’s article reflects her extensive experience with online teaching and learning. She demystifies learning assessment and, more importantly, extends it from theoretical realms with examples and suggestions, to make the concepts and applications readily accessible. And don’t let the “Online” in the title fool you; this useful article is equally applicable to face-to-face classrooms.

Mauri Collins
DEOSNEWS Editor

Evaluating Student Mastery and Program Effectiveness

Maggie McVay Lynch

Adapted from book chapter in Lynch, M. (2002). The Online Educator: Creating the Virtual Classroom. London: Routledge.

Evaluation of students in Web-based education should occur much as it does in traditional classroom settings. The key is determining how to do that when students are not physically present. It is equally important to evaluate the course effectiveness and the success of the entire Web-based program.

For over two decades, a dualistic debate has raged over whether assessment should focus on accountability or improvement. Today, most educators have come to accept that dealing with both is important. However, this article will concentrate on evaluation as being first and foremost about improving student learning and secondarily about determining accountability for the quality of learning produced.

Angelo (1999) discussed the challenges of student and program assessment. He proposed four pillars of transformative assessment to help evaluation move forward within a learning community of students and faculty involved in assessment. His four pillars are:

1. Build shared trust—Encouraging participants to share examples of successful teaching or assessment practices allows them to present their best face and demonstrates that each is a smart person with ideas to contribute.
   
2. Build shared motivation—Most people are more productive when working toward clear, personally meaningful, reasonable goals.
   
3. Build shared language and concepts—Develop a collective understanding of language and new concepts (mental models) for describing, manipulating, and meeting the goals.
   
4. Build shared research guidelines—Individual campuses and programs can benefit from constructing their own specific lists of principles or guidelines that serve as the criteria for evaluating their own assessment plans and efforts.
   

Angelo (1999) accurately summarized the purpose of assessment when he said: “If we plan and conduct our assessment projects at every step as if learning matters most—and not just student learning, but ours as well—then the distance between means and ends will be reduced and our chances for success increased.”

EVALUATING STUDENT MASTERY

Teachers have been evaluating students since formal education began. Student mastery may be assessed through a variety of methods including oral interviews, written tests, practical application of concepts and procedures, and asking students to teach the concept or skill to someone else. Unfortunately, both in traditional education and in Web-based education student, evaluation is often given short shrift when designing instruction. Usually this misconnection in evaluation occurs because teachers or course designers fail to create a direct relationship between instructional objectives and assessment measures. To establish this connection, three key ideas are crucial:

1. Obtain a good match between the type of objective you wish to measure (i.e., knowledge, skills, attitudes) and the means you use to measure it.
   
2. Use several data sources to gain as complete a picture as possible.
   
3. Remember that not all instructional objectives lend themselves to direct, precise measurement.
   

Table 1 provides examples of typical poor measurements for types of knowledge or skills and examples of a better measurement tactic.

In each of the above instances, the teachers or course designers thought they were measuring the objective. However, in each situation the measurement was at a lower level in Bloom’s (1956) taxonomy. The objectives were asking for the measurement of demonstrations, analysis, and synthesis, whereas the original measurement strategy was evaluating knowledge—particularly lower-level knowledge such as identification and recall. The final example tries to measure application through writing the program. However, the ability to construct a program does not measure the ability to analyze one that is already constructed.

Bloom’s (1956) taxonomy can be used to help phrase an objective so that you and the learner know what he or she is expected to be able to do after the lesson or the course. The levels indicate whether the learning involves the learner in lower- or higher-order thinking. There is no good or bad; however, you want to help students progress to the highest level they are able to achieve. Do not underestimate your students’ abilities in this area. Bereiter and Scardamalia (2000) found that even young children (aged 3–12) are capable of higher-order thinking skills earlier than previously thought. Among young children aged 3–10, they found clear evidence of children participating in both application and analysis. Children ages 10–12 also reached the higher-order skills of analysis and synthesis, with occasional evaluation. Keeping this mind, it seems teachers should work toward more teaching and mastery activities at these higher-order thinking skills levels both at a young age and certainly in adolescence and adulthood.

When determining how to measure mastery of these levels, it is beneficial to match the expected products that measure mastery with the level of the taxonomy. Table 2 gives some examples of potential measurement products at different levels in Bloom’s (1956) taxonomy.

As discussed above, the difficulty in accurately assessing student mastery lies in the accurate definition of learning objectives, as well as in the teaching and assessment of those objectives. Two decades ago, English (1978) illustrated this difficulty in his discussion of the “fictional curriculum.” This is the declared curriculum—what is assumed the student is learning. However, this may differ from the “real” or taught curriculum—that is, the curriculum as it is delivered to the student. It may also differ from the “tested” curriculum—what students actually learn. Finding tools to help in keeping the links between learning, teaching, and assessment is key to student success and to accurate measurement.

Another tool to use when planning any instructional design is to create a matrix to link outcomes, strategies and assessment. By placing your lessons into the matrix, you help to ensure that there is a direct relationship between your learning objectives and your teaching strategies. You can also easily determine if you are, in fact, assessing each learning objective. If you find you are not teaching or assessing an outcome, then you need to evaluate what to do. Is the outcome important? If so, add an additional teaching strategy and assessment. If not, delete the outcome from your list. It is possible to combine several objectives/outcomes under one teaching strategy or one assessment tool. Table 3 shows an example of two lessons, from an online Business Communications Course, placed in the matrix.

Note that the above matrix uses student assessments other than typical objective testing. This is due to learning outcomes that require more than the lower-order thinking skills of knowledge and comprehension. Also note that, in each case, the assessment tool or instrument measures more than one objective at a time. This helps to lessen the teacher workload for separate discrete measurement of each objective.

Another key factor in both of the examples in the matrix is the use of collaboration and the social construction of knowledge. Encouraging more cooperation and networking among peers has become important to many teachers today, and the Internet makes this possible. However, some instructors find difficulty in evaluating networking and collaboration in the Web-based environment. Networked projects are often dynamic, active, and constantly changing. They incorporate various activities by participants in different locations reacting or sharing at different times.

Ravitz (1997) and Reigeluth (1995) discuss the shift that takes place as education moves toward an information-sharing paradigm. This conversational learning method sometimes frightens educators new to the Web, as they feel a “loss of control” over the content and must deal with an added layer of complexity not usually encountered in the more controlled environment of a traditional lecture and test class.

Five examples of translating these assessment techniques to the online environment are discussed here: giving up control, reassessing outcome evaluation beyond testing, real-world application, project-based learning assessment, and student reflection as assessment.

Giving up Control

One of the most difficult aspects of teaching online is the inability to see the students. Teachers rely on “controlling” the classroom environment and thus controlling and evaluating student outcomes. For example, in a speech class, the course objective is usually for the student to be able to make effective, persuasive presentations. The unstated outcome is probably that the student will be able to do this even after leaving the class. The usual measurement of this objective relies on having the student give an oral presentation to the class. The instructor then evaluates the student’s presentation based on several criteria such as articulation, presence, body language, ability to answer questions, and audience control. The teacher may also ask the class to provide some peer critique of the speech. How, then, can this same evaluation and subsequent learning occur in an online course?

The first step is giving control of the student’s assessment to the student instead of to the instructor. This takes a great deal of trust of both yourself and the design of the course, as well as faith in the student’s ability to learn from assessment outside of your purview. First, you must believe that the online course has already presented the knowledge base needed for the student to be an effective presenter (e.g., information about how to stand, how to speak, how to gauge the audience). Second, you must provide students with an opportunity to test that knowledge. This might be done by having students do a presentation for a group, of their choice, in their home location (e.g., a local school, a civic organization, a church, their place of business, etc.). Give the students an evaluation criteria sheet—the same rubric you would use to evaluate the students in class—and ask them to have one or more audience members complete the evaluation following the speech. Finally, ask the student to use those evaluations to reflect on the speech and discuss what went well, what could be improved, and to rework the presentation for another iteration.

In giving up control of the evaluation, several things occurred in this example:

1. The student is given responsibility for learning and for evaluation.
   
2. The student learns to use resources outside of the teacher for ongoing assessment beyond the course.
   
3. The evaluation reflects a real-world environment instead of that in the classroom.
   
4. The student must use the higher-order thinking skills of application, analysis, synthesis, and evaluation in writing a reflection of the event.
   

In keeping with the tradition of “doing assessment as if learning matters most,” it seems that this type of assessment provides accurate reflection of the learning objective and best assists the student.

Knowledge Outcomes Evaluation

One of the unfortunate developments of Web-based education has been the increasing use of “objective” testing (e.g., multiple choice, true/false, fill-in-the-blank). Because of the speed for testing and feedback, it is a natural use of the online environment. Certainly, there is a place for this type of mastery testing—that is when assessing lower-order thinking skills such as knowledge or comprehension. Writing objective test questions that also measure higher-order thinking skills effectively is very difficult (and therefore this method is rarely used).

Furthermore, by relying solely on episodic objective testing (e.g., a mid-term and a final) of student mastery, you exclude several groups of skilled students: those who do poorly with this type of testing, those who learn over time, and those who learn best through actual experience. Learning is a complex process. It entails not only what students know, but what they can do with that knowledge. Also, it goes beyond knowledge to values, attitudes, and performance beyond the classroom. Thus, your assessment of student mastery should include a diverse array of methods.

A survey of United States colleges and universities found that methods for assessing students continue to be traditional, emphasizing quantitative instruments. Institutions do make limited use of more innovative—and usually more qualitative—assessment techniques such as portfolios, capstone projects, and observations of student performance. However, only 34% indicated they engaged in more complex assessment activities, such as collecting information about higher-order thinking skills, affective development, or professional skills. Even fewer (23%) asked students about their civic or social activities. Is it enough to rely on the easily quantifiable indicators of student mastery in the name of consistency? Or should we risk some inconsistency and seek to effectively measure mastery that will matter in the student’s transition from school to the real world?

Real-World Application

A mantra of constructivism theory has been “situated cognition”—the concept that students will learn more if the theory is presented in relation to real-world applications. This makes the learning relevant to the student’s situation. The Web-based learning environment provides a unique opportunity to capitalize on this situated-cognition approach to learning. As pointed out previously, you are already in a position of giving up control when teaching in the Web-based environment. One effective way to both give up control and provide a real-world learning experience is to require students to apply the new knowledge in their own environments.

For example, an online education graduate program at a Florida university requires that most homework assignments be applied in the real world. In the doctoral research class, instead of working statistical problems with case studies or common book examples, students are required to actually administer a survey in their work environments, gather and analyze the data, and write a report as their culminating assignment.

An undergraduate organizational communications course at an Ohio university requires students to analyze communications in their work place for one of the assignments, then make recommendations for improvement. A fourth-grade elementary school in Georgia requires students to demonstrate examples of physics that they experience in their own homes. All of these examples use situated cognition to teach, reinforce, and assess complex concepts.

Having students apply their learning to real-world situations demonstrates the use of several higher-order thinking skills: application, analysis, and synthesis. It also helps to meet the often unarticulated outcome of continuing to use the concepts, even after leaving the course.

Project-Based Learning Evaluation

Similar to real-world application of concepts, project-based learning provides analogous benefits. The examples of real-world evaluation given in the topic above are also examples of project-based learning. In each of those situations, the student is presented with a problem to analyze and resolve through making recommendations. This type of project-based learning evaluation requires synthesis of many concepts as well as evaluation and prioritizing of the concepts when selecting on which to report.

In addition to having students demonstrate mastery of skills and knowledge, project-based learning can also allow the teacher to assess the scaffolding of concepts. A complex project can be built that requires the student to follow a procedure in order to complete it. Then the teacher can evaluate where in the procedure problems may occur. Another type of project-based scaffolding evaluation would occur in a topic that requires prior knowledge to succeed. For example, a project that requires doing a trial-balance in account would also require the student to know how to extract information correctly from credit and debit journal items. The combination of a procedure and prior knowledge and understanding may be followed to see where in the learning the student may have difficulties.

Another type of scaffolding evaluation would be for conceptual development that is based on relationships that are not necessarily linear. For example, in an undergraduate business ethics course, an instructor assigned a project of creating a collage of pictures and words that represented the ethics concepts covered in the course. This type of project has the potential of revealing much more than a simple definition of ethics or any absolute reflection of right and wrong. The collage may reveal several levels of ethical dilemmas, a pictorial representation of affective domains relating to ethics, as well as a philosophy about ethics. The project requires multiple inputs and outputs to meet the assignment and thus accurately reflected the inherent individuality in ethics and values. Here evaluating student mastery through a project probably netted much more information than the traditional multiple-choice test or short essay.

The Use of Student Reflection Papers

Chinese philosophy teaches us about Yin and Yang. Too often, in the daily lives of many Westerners as well as in our approach to education, we concentrate only on the “yang”—the doing, the demonstrating, the creating. In fact, many educators completely ignore the “yin”—the reflection, the patient waiting, the quiet times. However, for many students, taking time for reflection (yin) is the best way for them to internalize knowledge and thus to retain it past the end of the course.

Teachers often mistakenly believe that there is no need to give time for reflection, as the students who need that will find a means to do that on their own. Instead, these instructors concentrate on how to pour more information into the perceived empty vessel of student minds. Unfortunately, this method is usually not successful. In the first place, students are not empty vessels awaiting the teacher’s knowledge. Rather, student minds are like ours, filled with all types of important information, special needs, and many distractions. Students, like the rest of us, also have busy lives filled with “yang,” and therefore they parse their time to meet only the immediate needs for learning at the moment. Even if they yearn for some reflective time, like us, students find it is hard to schedule reflection into their daily lives.

The assignment of reflection papers does two important things: (1) It allows (forces) the student to take time to reflect on their learning. (2) It provides the instructor with invaluable information about the students’ perceptions of the topic, the development of potential innovative concepts or uses based on their learning, and highlights any misconceptions. When required to engage in reflection during a course, students’ end-of-course evaluations often include comments such as, “I didn’t realize how much I learned until I did this assignment,” or “I learned so much more in this course because I was forced to take time and think about how it affected me.”

Using reflection papers for evaluation can come in several forms:

• Journals—Students chronicle their perceptions of learning, topic relevance, and applications throughout a course.
  
• Specific reflection papers—Students ponder a specific concept, dilemma, case study and write descriptions of their feelings as they work through the assignment.
  
• Beginning-of-course reflections—Students usually note their preconceived expectations of the course, the teacher, the content, their fears or interests in relation to the topic.
  
• End-of-course reflections—Students evaluate the impact of the course in their learning or their lives.
  
• Debriefing reflections—Students take a short period of time to record what went well or what went wrong following a specific learning event (i.e., a presentation, a role-play situation, a group project).
  

If we are truly using evaluation to make learning matter, then the use of reflection is a key ingredient in the complexity of student mastery.

Finally, to summarize student assessment, let us look at the American Association for Higher Education’s (2000) 9 Principles of Good Practice for Assessing Student Learning (used with permission):

1. The assessment of student learning begins with educational values. Assessment is not an end in itself but a vehicle for educational improvement. Its effective practice, then, begins with and enacts a vision of the kinds of learning we most value for students and strive to help them achieve. Educational values should drive not only what we choose to assess but also how we do so. Where questions about educational mission and values are skipped over, assessment threatens to be an exercise in measuring what’s easy, rather than a process of improving what we really care about.
   
2. Assessment is most effective when it reflects an understanding of learning as multidimensional, integrated, and revealed in performance over time. Learning is a complex process. It entails not only what students know but what they can do with what they know; it involves not only knowledge and abilities but values, attitudes, and habits of mind that affect both academic success and performance beyond the classroom. Assessment should reflect these understandings by employing a diverse array of methods, including those that call for actual performance, using them over time so as to reveal change, growth, and increasing degrees of integration. Such an approach aims for a more complete and accurate picture of learning, and therefore firmer bases for improving our students’ educational experience.
   
3. Assessment works best when the programs it seeks to improve have clear, explicitly stated purposes. Assessment is a goal-oriented process. It entails comparing educational performance with educational purposes and expectations—those derived from the institution’s mission, from faculty intentions in program and course design, and from knowledge of students’ own goals. Where program purposes lack specificity or agreement, assessment as a process pushes a campus toward clarity about where to aim and what standards to apply; assessment also prompts attention to where and how program goals will be taught and learned. Clear, shared, implementable goals are the cornerstone for assessment that is focused and useful.
   
4. Assessment requires attention to outcomes but also and equally to the experiences that lead to those outcomes. Information about outcomes is of high importance; where students “end up” matters greatly. But to improve outcomes, we need to know about student experience along the way—about the curricula, teaching, and kind of student effort that lead to particular outcomes. Assessment can help us understand which students learn best under what conditions; with such knowledge comes the capacity to improve the whole of their learning.
   
5. Assessment works best when it is ongoing not episodic. Assessment is a process whose power is cumulative. Though isolated, “one-shot” assessment can be better than none, improvement is best fostered when assessment entails a linked series of activities undertaken over time. This may mean tracking the process of individual students, or of cohorts of students; it may mean collecting the same examples of student performance or using the same instrument semester after semester. The point is to monitor progress toward intended goals in a spirit of continuous improvement. Along the way, the assessment process itself should be evaluated and refined in light of emerging insights.
   
6. Assessment fosters wider improvement when representatives from across the educational community are involved. Student learning is a campus-wide responsibility, and assessment is a way of enacting that responsibility. Thus, while assessment efforts may start small, the aim over time is to involve people from across the educational community. Faculty play an especially important role, but assessment’s questions can’t be fully addressed without participation by student-affairs educators, librarians, administrators, and students. Assessment may also involve individuals from beyond the campus (alumni/ae, trustees, employers) whose experience can enrich the sense of appropriate aims and standards for learning. Thus understood, assessment is not a task for small groups of experts but a collaborative activity; its aim is wider, better-informed attention to student learning by all parties with a stake in its improvement.
   
7. Assessment makes a difference when it begins with issues of use and illuminates questions that people really care about. Assessment recognizes the value of information in the process of improvement. But to be useful, information must be connected to issues or questions that people really care about. This implies assessment approaches that produce evidence that relevant parties will find credible, suggestive, and applicable to decisions that need to be made. It means thinking in advance about how the information will be used, and by whom. The point of assessment is not to gather data and return “results”; it is a process that starts with the questions of decision-makers, that involves them in the gathering and interpreting of data, and that informs and helps guide continuous improvement.
   
8. Assessment is most likely to lead to improvement when it is part of a larger set of conditions that promote change. Assessment alone changes little. Its greatest contribution comes on campuses where the quality of teaching and learning is visibly valued and worked at. On such campuses, the push to improve educational performance is a visible and primary goal of leadership; improving the quality of undergraduate education is central to the institution’s planning, budgeting, and personnel decisions. On such campuses, information about learning outcomes is seen as an integral part of decision making, and avidly sought.
   
9. Through assessment, educators meet responsibilities to students and to the public. There is a compelling public stake in education. As educators, we have a responsibility to the publics that support or depend on us to provide information about the ways in which our students meet goals and expectations. But that responsibility goes beyond the reporting of such information; our deeper obligation—to ourselves, our students, and society—is to improve. Those to whom educators are accountable have a corresponding obligation to support such attempts at improvement.
   

PROGRAM EVALUATION

With the surge in public demand for accountability and some concern over the viability of Web-based learning in terms of quality, it is important to have a good program evaluation in place. However, beyond the needs of political justification, it is essential that your program evaluation accurately informs changes in curriculum, teaching strategies, and student assessment.

Recognizing that good programs are made that way through constant vigilance and by learning from feedback received along the way, you will be faced with several preliminary decisions:

• What decisions do you want to make as a result of the evaluation?
  
• Are you primarily interested in program improvement or in justifying the existence, removal, or expansion of a program?
  
• Do you want to use an inside or outside evaluator?
  
• Do you want to use a quantitative or qualitative evaluation strategy?
  

Determining an evaluation strategy will be essential in fulfilling whatever your mission is for evaluation, or in answering the questions you have posed. For example, a quantitative strategy might look at factors such as numbers of enrollees, their ratings of program quality, the percentage of graduates, perceptions of persons dropping out or graduation from the program. A quantitative strategy might be most useful in taking program action such as expanding course offerings or adding a new degree program. A qualitative approach might examine profiles of students based on regional location qualities, how studying takes place, how leadership is exerted, how grades are obtained, and how resources (such as libraries and Internet access) outside of the university are used. A qualitative strategy is used to understand the learning process, what is going on, and how it might be influenced from within or from outside to provide a more satisfying educational experience.

Web-based program evaluation is usually divided into two areas: content and instructor evaluation. Certainly, it is difficult to separate these two elements as they are inextricably linked. However, for this article we will examine the necessity of evaluating these two areas separately and provide some sample student survey tools that may assist you in gathering some data.

Content Evaluation

Researchers often complain that one of the weakest elements in the design and development of Web-based education programs is failure to routinely assess the effectiveness of the materials and media. Evaluation should be practiced continuously through the design, development, and implementation cycles to ensure that things work as anticipated and intended.
Moore and Kearsley (1996) identified two key criteria of content delivery evaluation: data collection methods and measures. Table 4 looks at these two categories and describes methods that can be used to evaluate Web-based instruction.

Each technique has strengths and weaknesses. To get the best results, it is recommended that more than one technique and several measures be used to obtain a complete picture of how well a course or program is working. Evaluation experts also recommend that a neutral party conduct the evaluation—that is, someone who is not part of the course or program design team.

Moore and Kearsley (1996) identified twelve general principles to consider when evaluating any course design:

1. Good structure—Course materials must be well defined and display internal consistency among different parts of the course.
   
2. Clear objectives—Identify suitable learning experiences and subsequent evaluation.
   
3. Small unit—The content and course organization should be presented in small units, preferably that correspond to a single instructional objective or learning activity.
   
4. Planned participation—Opportunities for student interaction should be embedded throughout the course materials.
   
5. Completeness—Extensive commentary or examples should be provided.
   
6. Repetition—Important ideas are reinforced to compensate for distractions and memory limitations.
   
7. Synthesis—Important ideas are woven together (usually in summaries).
   
8. Stimulation—Materials capture and hold the attention of students through varied formats, content, or guest participation.
   
9. Variety—Format and media variety are present to appeal to student interests, backgrounds, and learning styles.
   
10. Open-ended—Assignments, examples, and problems allow students to adapt the content to their own situations.
    
11. Feedback—Regular feedback is provided on assignments and student progress in the course.
    
12. Continuous evaluation—Effectiveness of materials, media, and instructional strategies are routinely assessed using a variety of methods.
    

The use of student surveys is one method of gathering summative data for evaluation. When developing this type of tool, however, it is important to ensure the students understand the separation of the course content (materials, assignments, activities) from the instructor’s participation, personality, or additions. It is also important to determine, in advance, what aspects of the course delivery are important or desired. In other words, you need to have a concept of what constitutes a “good” course before attempting to develop an effective evaluation instrument. Does your concept include the use of images, specific interaction requirements, or a variety of learning styles? Whatever you determine to be consistently important from one course to the next should be included in your survey instrument. One example of an online course evaluation instrument used successfully is presented in Appendix

Instructor Evaluation

For the individual teacher conducting a Web-based course, the indicators of how the class is going are in many ways similar to on-campus instruction. Some sources of information to help shape and maintain quality in a course are:

• e-mail messages from students;
  
• student verbal or written feedback about the class in reflection assignments;
  
• student debriefing comments or sentiments, favorable or unfavorable, following specific activities;
  
• end-of-course instructor evaluations.
  

A sticky subject at most schools is the evaluation of the instructor. In the university system, end-of-course student evaluations often serve for promotion and tenure purposes. Consequently, the creation, validating, and reliability of any instruments used for this purpose is of high concern to faculty.

As with the course evaluation, it is important to identify in advance what aspects of online course facilitation are considered key success factors. For example, is it important for the instructor to be in regular contact with the students? Is the instructor expected to develop, on the fly, additional Web-based materials as students need them? How much interaction outside of e-mail (e.g., chat room participation, bulletin board posting, specific electronic comments on homework) is expected of a good instructor? How much of the instructor’s enthusiasm for the topic is expected? All of these and many more aspects of Web-based course delivery need to be weighed and prioritized prior to developing an instrument.

The sample instructor evaluation instrument in Appendix B is one that has been used successfully.

In summary, evaluators of Web-based instruction have an opportunity to explore educational applications using a new and evolving medium. This work requires the consideration of appropriate research methods and assessment strategies. As with any evaluation design, it is important to establish outcomes for the evaluation. These outcomes will then determine what constitutes student mastery, content excellence, or instructor effectiveness.

References

American Association for Higher Education. 2000. [online]. 9 principles of good practice for assessing student learning. Available at http://www.aahe.org/assessment/principl.htm

Angelo, T. 1999. [online]. Doing assessment as if learning matters most. AAHE Bulletin. Available at http://www.aah.org/Bulletin/angelomay99.htm

Bereiter, C., and M. Scardamalia. 2000. [online]. Beyond Bloom’s taxonomy: Rethinking knowledge for the knowledge age. Ontario Institute for Studies in Education, University of Toronto. Available at http://www.csile.oise.utoronto.ca/abstracts/Piaget.html

Bloom, B. S., ed. 1956. Taxonomy of educational objectives: The classification of educational goals. Handbook I. Cognitive Domain. New York: Longman.

English, F. 1978. Quality Control in Curriculum Development. Arlington, VA: American Association of School Administrators.

Moore, M., and G. Kearsley. 1996. Distance education: A systems view. Belmont, CA: Wadsworth Publishing.

Ravitz, J. 1997. Evaluating learning networks: A special challenge for Web-based instruction. In B. Khan (Ed.) Web-based Instruction. Englewood Cliffs, NJ: Educational Technology Publications.

Reigeluth, C. 1995. The imperative for systemic change. In Systemic Change in Education, ed. C. M. Reigeluth and R. Garfinkle. Englewood Cliffs, NJ: Educational Technology Publications.

Author Information

Maggie McVay Lynch, Ed.D.
Portland State University
SMC 18F, Instruction and Research Services
PO Box 751
Portland, OR 97207-0751

Phone: 503-725-9116
Fax: 503-725-3476
E-mail: mmlynch@odin.pdx.edu
http://web.pdx.edu/~mmlynch