DEOSNEWS Vol. 7 No. 8, ISSN 1062-9416.

Copyright 1997 DEOS - The Distance Education Online Symposium.

- - - - - - - - - - - - - - - - - - - - - - - - - -

EDITORIAL

One element in the success of distance education is the systematic evaluation of programs throughout the design, development, and implementation process. Evaluations of learner attitudes and satisfaction are common in the literature and can provide valuable input to the development team with regard to the effectiveness of course materials. With the increased use of video programming to deliver education and the growth in the number of adult students in higher education, evaluations may also need to consider whether the materials being produced are appropriate for the adult student audience.

The article in this issue of DEOSNEWS describes a formative evaluation of a sample video program from the telecourse series, _Nutrition Pathways_. The evaluation was designed to determine if the sample video was appropriate for the adult student population for which it was intended. Formative evaluation criteria were identified, based on the concepts of adult education principles. These criteria were then incorporated into a questionnaire which was administered during the development stage of the video production to students enrolled in an introductory nutrition course. The results of the study suggest that the sample video program met the evaluation criteria and that the instructional strategies used in the sample video adhered to adult education principles.

===========================

FORMATIVE EVALUATION OF A SAMPLE VIDEO PROGRAM

FROM THE _NUTRITION PATHWAYS_ TELECOURSE

BASED ON ADULT EDUCATION PRINCIPLES

Nora Coto Busby

R. Jan LeCroy Center for Educational Telecommunications

INTRODUCTION

Production of nationally distributed telecourses is a two-year process that costs approximately $1 million. A telecourse usually consists of 26 thirty-minute video programs, an adopted textbook, a student study guide, a faculty guide, test bank, and ancillary materials such as software or multimedia programs. The most costly and time-consuming component of production is the video programs. Students' reactions to the video programs are not received until after the product is produced and distributed. At this point, revisions are too costly to justify making them. As a consequence, students may be required to watch video programs that they consider inappropriate, uninteresting, or even offensive over a program shelf life of seven to ten years.

The purpose of this study was to determine whether the video programs for the distance education telecourse _Nutrition Pathways_ were appropriate for the adult student population for which they were designed. The evaluation occurred during development of the video programs and was based on the extent to which a sample program from the series met the principles of adult education as judged by students.

REVIEW OF THE LITERATURE

Adult Education Principles

No single theory of learning has been established for adults (Merriam and Caffarella 1991). As a consequence, there is no comprehensive model on which to base decisions regarding instructional strategies to use in educational or training programs for adults. The purpose of this review was to identify common assumptions in both adult learning theory and practical applications that could be used as a conceptual base of information regarding adult education principles and that could be used to evaluate instructional strategies selected for adult learners.

The following five common principles were identified: 1) the learning must be relevant to the learners; 2) the learners must be motivated to learn; 3) the instruction must incorporate varied strategies that tap into the learners' experience base; 4) the learners must feel a sense of control over their own learning; and 5) instructional strategies must accommodate the cognitive and/or physiological needs of the learners.

Knowles (1990) was one of the first to incorporate the notion that adults need to know why they are learning something in terms of its relevance to life-, task-, or problem-centered issues. If adults do not perceive the relevance, importance, or value of the learning task, they will not be committed to learn. A connection between relevance of learning and motivation or readiness to learn is postulated as a requirement for learner motivation in Keller's ARCS (Attention, Relevance, Confidence, and Satisfaction) model of motivation (Gagne and Perkins 1988). Adults are motivated to learn things that are relevant to them, what "...they need to know and be able to do in order to cope effectively with their real-life situations" (Knowles 1990, 60).

The third principle deals with individual differences among adult learners, which are due to age and experience. Adult learning processes are different and have more to do with transformation of experience than accumulation of knowledge (Merriam and Caffarella 1991). This point of view is supported by constructivist theory, which defines learning as something that creates meaning through direct, individual experiences with the environment (Bednar, Cummingham, Duffy, and Perry 1991, cited in Ertner and Newby 1993). Learners' personal knowledge bases and interpretations of the world are based on individual experiences and interactions (Ertner and Newby 1993).

If adult learners are motivated by relevant learning and benefit from incorporating their own experiences into the learning, it is not surprising that they want some degree of control over their own learning. Knowles (1990) develops this assumption in the form of the learners' self-concept. The self-concept of being an adult-being responsible for oneself and one's own lifeincludes the need for respect and recognition as being capable of self-direction. When others attempt to impose their will, adults resist. Constructivists share this viewpoint (Holmes and Leitzel 1993). Constructivists believe that learning is a personal process based on the learner's own conceptions and knowledge, which shifts the emphasis of learning to provision of choices among multiple methods, approaches, and instances in which the new learning is actively used. The learner decides how to learn.

A question that begs asking is, "What are the cognitive and physiological conditions of adult learners that may affect learning?" Studies have shown that an adult's response time slows with age, and time limits and pressures have a negative effect on learning performance (Merriam and Caffarella 1991). In addition, adults are more likely than younger learners to have health problems, be fatigued, require medication, have disuse of some abilities, experience interference from previous learning, and have impaired short-term memory, for example (Merriam and Caffarella 1991).

Although no comprehensive model of adult learning exists, the principles derived from adult learning research and practice can be used to develop criteria for selecting instructional strategies for adults. Applied research on the effectiveness of such strategies based on these adult learning principles can be performed and the results used to improve or modify the instructional strategies selected.

Formative Evaluation

There is general consensus in the literature that formative evaluation is the collection of information during the development of instructional materials for the purpose of improving instruction (Boutin and Chinien 1992; Chinien 1990; Dehar et al. 1993; Dick and Carey 1990; Dick and King 1994; Flagg 1990; McAlpine 1992). In 1967, Scriven coined the term and emphasized the need for developers to "...conduct non-judgmental formative evaluations with students in order to improve the instruction prior to its publication" (Dick and King 1994, 3). With formative evaluation, materials can be modified and improved before production is complete.

Although Scriven did not delineate procedures for conducting formative evaluations, most formative evaluation studies involve review and validation of materials by experts, review or tryout by students during development, and field testing in the environment in which the materials will be used (Chinien 1990; Dehar et al. 1993; Dick and Carey 1990; Dick and King 1994; Flagg 1990). Student review or tryout typically involves collection of observation, performance, and attitudinal data (Dick and Carey 1990; Dick and King 1994; Flagg 1990; McAlpine 1992). The questions used to obtain student attitudinal data vary but generally include whether the material was clear, readily understood, motivating, and relevant; whether the examples and practice provided were helpful; and whether the content was satisfying and useful (Dick and Carey 1990; Dick and King 1994; Flagg 1990).

Evaluation Criteria for Instructional Video

Recent literature on formative evaluation of instructional video contains minimal discussion about the basis for selecting student attitudinal questions. Sumner (1991) describes the formative evaluation conducted on BBC educational broadcasts as piloting early programs with groups of students and obtaining comments from providers. No further details are provided on how broadcasts were piloted or what questions were asked of providers. Tomic (1994) describes the formative evaluation conducted on a prototype psychology course as based on "...whether, and to what extent, the target group actually achieved the learning objectives" (1203). Apparently, students were not asked about their attitude toward the instructional programs.

Flagg (1987) conducted the most extensive and descriptive formative evaluation study, in which formative evaluation of a pilot telecourse video program involved assessing "...the appeal and comprehensibility of program segments as well as to measure student short-term recall" (8). In none of these studies were adult learning principles used as the criteria on which to judge the pilot programs, even though all studies were targeted to adult student populations.

Adult learning principles appear to some extent in instructional video evaluation instruments designed for use by postproduction reviewers. The most recent evaluation instrument identified was produced by Beaudin and Quick (1993), who developed their criteria by compiling suggestions for quality indicators for videos and "...synthesizing the instructional design methodology of Brookfield (1986), Friere (1970), Galbraith (1991, 1992), and Seels and Glasgow (1990)" (Beaudin and Quick 1993, 5). As in many other video evaluation instruments, their list of adult learning criteria is not comprehensive, nor are adult learning criteria placed in a separate, distinguishable category.

The most comprehensive work on development of criteria and questions for use in evaluating telecourses and component video programs was done by Lane (1989). The model and evaluation instrument was developed and validated by a sample of "...administrators, instructors, producers, directors, writers, instructional designers, distributors, consortia members, and authors" (48) from Canada, Great Britain, and the United States. The only distinguishable adult learning criterion included for the video component of a telecourse was whether the video enriched learning with real-life applications in realistic settings and field trips to realistic locations.

METHODOLOGY AND PROCEDURES

Evaluation Criteria

From the literature review, a conceptual base of information was established regarding adult education principles and the criteria and models used in similar studies. The initial draft was reviewed by an expert review panel consisting of the content specialist, producer, and instructional designer of the _Nutrition Pathways_ production team. The panel modified the first draft and then prepared a final draft of eleven criteria (Table 1). The formative evaluation criteria were then incorporated into an eleven-item Likert-type questionnaire with six levels of agreement from completely agree to completely disagree for each item (Table 2).

Sample Video Program

A ten-minute sample of a video program being developed for the _Nutrition Pathways_ telecourse was produced for the study. The program, "Adulthood and Aging," contained three segments or profiles, each focusing on a different stage of aging. The segment in the video sample profiled a couple in their thirties and the effect of nutrition and lifestyle on their aging process. The ten-minute sample contained the instructional strategies selected for the entire series. The scenario used real people in their own life situation to illustrate the concepts. Voiceover narration and on-camera experts were employed throughout the video to establish the context, elaborate on content, clarify issues, and relate the material to other situations and applications.

Subjects

The subjects of this study were Dallas County Community College (DCCCD) students enrolled in the spring 1996 semester "Introduction to Nutrition" traditional classroom course and their instructors. The courses were offered in the morning, late afternoon, and evening, and on the weekend. Six of the seven campuses in the DCCCD offered "Introduction to Nutrition" during the spring 1996 semester. All students self-selected into each of the classes. There was a total of ten classes, one hundred eighty students, and nine instructors. One instructor taught two classes. Students who enroll in the "Introduction to Nutrition" course are either nutrition and allied health majors who are required to take a nutrition course or nonhealth majors who take the course as an elective. The _Nutrition Pathways_ telecourse is targeted to both of these student groups.

Treatment.

The sample video program was shown to students and instructors during one of the scheduled nutrition classes. Permission was obtained from each instructor by telephone in advance. The content specialist or instructional designer gave a brief introduction to the telecourse, explained the purpose of the study, and showed the sample video program. After viewing the video, the students and instructor were asked to complete the Likert-type questionnaire and indicate sex, age, race/ethnicity, and student status.

Data Analysis

For each of the eleven items, the number of students from all classes who marked each of the six levels of agreement was totaled and percentages were calculated. In addition, the characteristics of all subjects were totaled by category and percentages calculated. Means and standard deviations were calculated for each item by campus. A one-way analysis of variance (ANOVA) was used to derive group means and group standard deviations. A posthoc comparison of means for unequal sample sizes using Tukey's Standardized Range Test (HSD) was then used to determine which, if any, pairs of group means were significantly different.

During data analysis, an error was discovered on the questionnaires given to students at one college. Item 6 on the Brookhaven College questionnaires was different from Item 6 on all other campus surveys. As a result, the Brookhaven scores were not included in any of the calculations for Item 6.

Limitations

Experimenter and subject effects may have posed a limitation on the internal and external validity of the study. Before viewing the sample video program, the instructor and students were told that they were participating in a study and that the experimenter was either the content specialist or the instructional designer of the telecourse. In addition, the instructor's reception to the study may have influenced how his or her students responded.

The use of only one video sample from the series posed additional limitations. The real-life situation scenario selected for the sample was only one of three selected for "Adulthood and Aging." The three scenarios focused on adults at different stages of aging. Subjects made judgments regarding the aging process after viewing only one scenario.

RESULTS

The number (frequencies) and percentages of students from all classes who marked each of the six levels of agreement for each of the eleven items are shown in Table 3. In general, over 85% of the subjects marked completely agree, mostly agree, or somewhat agree (levels 1, 2, or 3) for all items, meaning over 85% of the subjects agreed that the video sample met each of the evaluation criteria. Over 58% of all subjects marked either completely agree or mostly agree (levels 1 and 2) for all items. More than 5% of the subjects marked four items either completely disagree or mostly disagree (levels 5 and 6). The subjects marked either level 5 or 6 for Item 3 (7.6%), Item 6 (5.4%), Item 9 (6%), and Item 11 (7.2%).

The characteristics of all subjects totaled by category and calculated by percentages are presented in Table 4. The subjects were predominantly white women aged 21 through 39. Approximately the same number and percentage of subjects were under 21 (32, 16.7%) as were over 40 (21, 10.9%). The percentages of African American and Asian subjects were relatively the same at 10.9% and 11.4%, respectively, with lower percentages of Hispanic subjects (8.1%). In addition, 17 of the 180 students were graduate students.

Means and standard deviations for each item by campus were calculated and used to derive the group means and group standard deviations shown in Table 5. From the group data, a posthoc comparison of means for unequal sample sizes using Tukey's test (HSD) revealed no significant pairwise differences among the means (p>.001). That is, each campus had basically the same response to each evaluation criterion. No single campus rated any of the items differently than any of the other campuses. The campus groups were homogeneous.

DISCUSSION

The results of this study indicate that a vast majority of the subjects (85%) generally agreed that the sample video program met the evaluation criteria. In addition, this general agreement was held by all campuses equally. Because the Brookhaven College group means were not significantly different from any other college group means on Items 1 through 5 and Items 7 through 11, there is a high probability that the results would have been the same on Item 6 if the Brookhaven scores had been correctly obtained.

The characteristics of the study subjects indicate that students enrolled in the "Introduction to Nutrition" courses were very similar. Typically, each campus in the DCCCD has a distinctly different student population. It was assumed that the student population differences across campuses would be prevalent across the student groups used in the study, hence the rationale for comparing means and standard deviations for each item by campus. Results from the study indicate that, in general, the student groups of the various campuses were more representative of the population of students who take nutrition courses throughout the DCCCD than of the student population of each campus. On the basis of these findings, it is recommended that future studies conduct group comparisons based on specific student variables such as age and gender in addition to such variables as campus location.

The evaluation criteria used in the study were derived from the adult education literature and evaluation instruments designed for evaluating educational programs. Although there is no one theory of adult learning, there are well established assumptions or principles regarding adult learning and education that can be transformed into evaluation criteria and used in developing adult education products.

There are also well-established views regarding the purpose and value of formative evaluation. Less clear is the basis for development of student attitudinal questions used during student review or tryout. Considering the growing number of adult learners enrolled in higher education and distance education programs, it is useful to incorporate into formative evaluation instruments a coherent set of criteria based on adult education principles.

For producers of distance education programs, the importance of conducting student formative evaluations to determine the extent to which instructional strategies selected for a particular program adhere to adult education principles can be extremely valuable. This is especially true for academic producers. Waiting until a program is produced and distributed to realize market appeal is fiscally risky, especially during a time of shrinking financial support and growing competition in distance education from both public and private sectors.

CONCLUSIONS

Within the study limitations, it can be concluded that the instructional strategies used in the sample video program adhered to adult education principles as judged by students. Because the instructional strategies used in the sample video program were representative of those selected for the series, it is further concluded that the instructional strategies selected for the video programs of the _Nutrition Pathways_ telecourse adhered to adult education principles as judged by students.

REFERENCES

Beaudin, B. P., and D. Quick. 1993. Instructional Video Evaluation. Report No. AEP-93-05. Fort Collins, CO: Colorado State University, School of Occupational and Educational Studies.

Boutin, F., and C. A. Chinien. 1992. Synthesis of research on student selection criteria in formative evaluation. Educational Technology 32(8):28-31.

Brookfield, S. D. 1986. Understanding and Facilitating Adult Learning. San Francisco, CA: Jossey-Bass.

Chinien, C. 1990. Paradigms of inquiry for research in formative evaluation. Performance & Instruction 29(9):26-29.

Dehar, M. A., S. Casswell, and P. Duignan. 1993. Formative and process evaluation of health promotion and disease prevention programs. Evaluation Review 17(2):204-220.

Dick, W., and D. King. 1994. Formative evaluation in the performance context. Performance & Instruction 33(9):3-8.

Dick, W., and L. Carey. 1990. The Systematic Design of Instruction. 3rd ed. New York: Harper Collins.

Ertner, P. A., and T. J. Newby. 1993. Behaviorism, cognitivism, constructivism: Comparing critical features from a design perspective. Performance Improvement Quarterly 6(4):50-72.

Flagg, B. N. 1990. Formative Evaluation for Educational Technologies. Hillsdale, NJ: Lawrence Erlbaum Associates, Inc.

Flagg, B. N. 1987. Formative evaluation in the development phases of a college telecourse. Paper presented at the annual meeting of the American Evaluation Association, October, Boston, MA.

Friere, P. 1970. Pedagogy of the Oppressed. M. B. Ramos, Trans. New York: Herder and Herder. (Original work published 1968).

Gagne, R. M., and M. P. Perkins. 1988. Essentials of Learning for Instruction. Englewood Cliffs, NJ: Prentice Hall.

Galbraith, M. W. 1991. The adult learning transactional process. In Facilitating Adult Learning: A Transactional Process, ed. M. W. Galbraith, 1-32. Malabar, FL: Krieger.

Galbraith, M. W. 1992. Nine principles of good facilitation. Adult Learning 3(6):10-11, 20.

Holmes, G. A., and T. C. Leitzel. 1993. Evaluating learning through a constructivist paradigm. Performance & Instruction 32(8):28-30.

Knowles, M. 1990. The Adult Learner: A Neglected Species. Houston: Gulf Publishing.

Lane, C. 1989. A selection model and pre-adoption evaluation instrument for video programs. The American Journal of Distance Education 3(3):46-57.

McAlpine, L. 1992. Highlighting formative evaluation: An instructional design model derived from practice. Performance & Instruction 31(10):16-18.

Merriam, S. B., and R. S. Caffarella. 1991. Learning in Adulthood. San Francisco: Jossey-Bass.

Seels, B., and Z Glasgow. 1990. Exercises in Instruction Design. Columbus, OH: Merrill.

Sumner, H. 1991. BBC education: The effectiveness and evaluation of educational broadcasting. Journal of Educational Television 17(3):159-172.

Tomic, W. 1994. Evaluation of a prototype psychology course for adult education. Psychological Reports 75(3):1203-1206.

NOTE: Tables are formatted to 4 inches. If they are unreadable, please contact the author for clarification.

TABLE 1. Evaluation Criteria

_____________________________________

- Motivating

- Builds on experience base of adults

- Uses realistic life situations

- Allows for learner control

- Presented on an adult level

- Promotes information integration

- Provides guidance and coaching

- Not overtaxing physically

- Not overtaxing mentally

- Relevant

- Promotes transfer of learning

================================

TABLE 2. Evaluation Questions for the

10-Minute Sample Video Lesson

_______________________________________

The topic was presented in a way that motivated me to want

to learn the material.

The topic was developed by building on the experiences of the

subjects who were profiled.

The life situation of the subjects profiled was realistic.

The topic was presented as though I was in charge of my

own learning.

The program was presented on an adult level.

It was easy for me to integrate the new knowledge I gained from

the program into what I already knew about the aging process.

The on-camera experts guided and coached me through the

learning process rather than directing me through it.

The amount of time spent on the topic did not overtax me

physically (ie, fatigue, headache, eye strain).

The amount of time spent on the topic was appropriate for the

amount of mental energy required to learn the topic.

Dealing with the aging process is a problem I am facing now

or will face in the future.

After viewing the 10-minute sample video lesson, I feel that I

can begin setting my own goals for improving my aging process.

==============================

Table 3. Frequencies and Percentages for

Each Item by All Classes

___________________________________________

Levels of Agreement

Item 1 2 3 4 5 6

______________________________________________

Item 1 51 66 44 15 7 2

(n=185) 27.6 35.7 23.8 8.1 3.8 1.1

Item 2 63 77 32 7 3 2

(n=184) 34.2 41.9 17.4 3.8 1.6 1.1

Item 3 70 58 31 10 7 7

(n=183) 38.3 31.7 16.9 5.5 3.8 3.8

Item 4 50 68 44 16 4 1

(n=183) 27.3 37.2 24.0 8.7 2.2 .6

Item 5 82 63 26 10 2 2

(n=185) 44.3 34.1 14.1 5.4 1.1 1.1

*Item 6 47 45 26 7 6 1

(n=132) 35.6 34.1 19.7 5.3 4.6 .8

Item 7 50 69 44 13 6 1

(n=183) 27.3 37.7 24.0 7.1 3.3 .6

Item 8 102 55 21 3 1 2

(n=184) 55.4 29.9 11.4 1.6 .5 1.1

Item 9 78 48 32 15 5 6

(n=184) 42.4 26.1 17.4 8.2 2.7 3.3

Item 10 96 43 28 9 4 4

(n=184) 52.2 23.4 15.2 4.9 2.2 2.2

Item 11 55 51 51 10 8 5

(n=180) 30.1 28.3 28.3 5.6 4.4 2.8

*Brookhaven College not included.

1 = Completely Agree; 6 = Completely Disagree

Note: Not all respondents answered all items. Numbers are rounded off.

================================

Table 4. Characteristics of All Subjects

_______________________________________

Characteristic Frequency Percent

_______________________________________

Sex

-Female 151 77.8

-Male 43 22.2

Age

-Under 21 32 16.7

-21-29 81 42.2

-30-39 55 28.6

-40-55 21 10.9

-Over 55 3 1.6

Race/Ethnicity

-African American 20 10.9

-Asian 21 11.4

-Caucasian 121 65.4

-Hispanic 15 8.1

-Other Ethnic 8 4.3

Status

-Undergraduate Student 163 86.2

-Graduate Student 17 9.0

-Faculty 9 4.8

================================

Table 5. Group Means and Standard Deviations

for Each Item

______________________________________

Item Group Mean Group SD

Item 1 2.23 .29

Item 2 1.95 .24

Item 3 2.13 .31

Item 4 2.19 .29

Item 5 1.88 .25

Item 6 2.06 .41

Item 7 2.28 .31

Item 8 1.67 .16

Item 9 2.10 .35

Item 10 1.88 .13

Item 11 2.27 .30

Note: Brookhaven College was not included in the group mean for Item 6.

===============================

Top of Page

The American Center for the Study of Distance Education (ACSDE)
The Pennsylvania State University
College of Education
110 Rackley Building
University Park, PA 16802-3202
Phone (814) 863-3764  FAX (814) 865-5878
ACSDE@psu.edu
www.ed.psu.edu/ACSDE

©2001 The Pennsylvania State University
College of Education