10:1  An Evaluation of Distance Learning in the University of Minnesota Human Resource Development Program.

EDITORIAL

This month's article is a cautionary tale backed up by a sound and comprehensive program evaluation by the University of Minnesota Human Resources Program. All the relevant stakeholders in the program - students, faculty, and project administration - were surveyed or interviewed and overall satisfaction with the program was high. Also taken into account were the "real costs" in infrastructure, administration and time. By all measures the program was a "success." The decision was made NOT to continue the program. The reason: it was not cost effective, given the number of students (and potential students) served.

Many institutions are rushing into the distance education market without a clear understanding of not only the set up costs, but the real costs of maintaining the distance delivery of a program once initiated. The University of Minnesota Human Resource Development Program discovered that unless the number of students served is very high, eventually someone discovers that the delivery of courses at a distance is not necessarily a cost saver, nor a money-maker for the institution. This critical fact is often overlooked in institution's enthusiasm to tap "new and unexplored markets."

Mauri Collins

DEOSNEWS EDITOR

AN EVALUATION OF DISTANCE LEARNING IN THE UNIVERSITY OF MINNESOTA HUMAN RESOURCE DEVELOPMENT PROGRAM

Teresa M. Tillson, University of Minnesota

409 Woodley St. West

Northfield, MN 55057

507-663-1124

tillson@microassist.com

James C. Warner, University of Minnesota

4216 44^th Street West

Minneapolis, MN 55424

Gary N. McLean, University of Minnesota

Dept of Work, Community and Family Education

College of Education

420 Votech Building

1954 Buford Ave.

St. Paul, MN 55108

ABSTRACT

The University of Minnesota’s Human Resource Development Distance Learning Project recently finished the final year of a three-year funding cycle. This paper evaluates the project, measuring its success from the viewpoint of students, faculty, and project administration. Suggestions are made to shape the Project’s strategies for the next three to five years and add this project’s experiences to the emerging literature on the effective uses of distance learning technology using electronic media.

INTRODUCTION

The Human Resource Development Distance Learning Project of the University of Minnesota was a pilot project to experiment with emerging technologies that allow students and faculty to interact across time and distance. It was funded by the College of Education and Human Development, University College/Continuing Education and Extension, and the Minnesota Extension Services. Established in Spring, 1995, the Project’s activities included classes offered via interactive television (ITV) and the Internet. ITV classes were broadcast from the St. Paul campus to University of Minnesota sites in Morris, Crookston, and Rochester, MN. Students could earn a Masters of Education degree entirely through distance learning, with the exception of one required course associated with the annual conference of the Academy of Human Resource Development (AHRD), Emerging Research in HRD. This course was designed to provide distance learning students with in-person contact with at least one senior professor, other students, and professionals in HRD. The first graduates of this program completed their degrees in Spring, 1997. In total, 131 different students took at least one class using some form of distance learning.

For the first two years of the pilot project, the only Internet course offered was "Training on the Internet," taught from the Pennsylvania State University, under contract with the University of Minnesota. This course used the tools available for web design and Internet training delivery. In the third year, six additional courses (all 1 and 2 credits) were offered via Internet by University of Minnesota faculty.

Faculty for the ITV courses were drawn from regular University of Minnesota faculty and instructors. Usually one course per quarter from among the regular HRD/Adult Education offerings was selected for ITV broadcast. Offerings were selected so students at the remote sites would be able to complete the necessary classes to complete an M.Ed. degree.

THE ISSUES

Distance learning using electronic media is a technology that is generating great interest in the HRD field. It offers the promise of bringing together students and instructors across physical boundaries when a traditional classroom experience would not be feasible. Additionally, many look for lowered costs due to decreased travel. However, distance learning’s effective use makes particular demands on the instructor/curriculum designer, on the student, and on the administration, which may differ greatly from the expected and well-known demands of the classroom experience. Moreover, student and faculty satisfaction with the distance learning experience may be affected by long acquaintance with and preference for face-to-face classroom interaction. Factors that affect faculty and student satisfaction with the learning experience are of special interest in this evaluation. Specific administrative costs and challenges are associated with distance learning, not the least of which is making sure all students and faculty have access to adequate and compatible equipment. Obtaining the needed equipment can make distance learning prohibitively expensive.

Finally, instruction via distance learning must be cost effective in order to be a viable alternative to classroom instruction. Marketing may need to be conducted differently in order to attract students who may be interested in seeking instruction through alternative methods. These students may not be aware that classes suitable to their needs are available and would not think to look for them in university catalogues. Candidates for instruction via distance learning include those who live too far from the university to attend classes and professionals and others whose business or personal schedules make regular classroom attendance difficult or impossible.

BACKGROUND

The Distance Learning Experience

While distance learning (DL) has its roots in correspondence education, there are additional issues for distance learning when electronic technologies are incorporated into its implementation. These issues are related to the structure of the learning experience, its content, the environment used to deliver the learning experience, and how the experience is administered (Kelly, 1990). These, in turn, influence the level of learning and attitudes toward the learning experience by the students, instructors, and institution.

Like other vehicles for delivering education, the structure of instruction via distance learning must incorporate appropriate instructional events for the type of learning which is to be attained (Winn, 1990). These events include the use of behavioral models and learning environments which support cognitive development. The use of a specific model should be determined by the instructional goals of the program (Bell & Davis, 1995, 1996), including the teaching of critical thinking skills (Newman, Johnson, Webb, & Cochrane, 1997) and group participation skills (Phillips, Santoro, & Kuehn, 1989).

Learning can be related to the degree to which any two-way communication transaction between instructor and student or among students takes place as part of the educational process in any learning environment (Shale & Garrison, 1990). Because distance learning takes place with a spatial or chronological separation between instructor and student and among students, technology is used to facilitate the two-way communication process. McIssac and Gunawardena (1996) cited the constructs of transactional distance and level of interaction as being important to effective communication.

The implementation support structure for distance learning "can be enormous" (Oblinger & Maruyama, 1996). In addition to development of the content and materials, support includes installation and operation of the technology delivery system, changes to the organization, and the responsibility for universal access. Technology for distance learning often includes instructional television networks and wide-area and local-area computer networks, including computer-mediated communication (Romiszowski & Mason, 1996). Kelly (1990) categorized these issues into planning, course preparation, and ongoing support. In addition to planning, preparing, and maintaining the technology infrastructure, organizational infrastructure changes include processes for planning and approving the courses and administering such student services as registration, communication, monitoring progress, and resolving problems. Often there is resistance or inability of institutional members to make the changes necessary for effective delivery of learning (Miller, 1997; Oblinger & Maruyama, 1996). The degree to which learning takes place and students’ administrative needs are addressed relates to the students’ attitudes toward the learning experience. The attitude of organization members toward distance learning may also affect student attitude. While media are used to present instructional messages that change attitudes, the media themselves can change attitudes as well (Simonson & Maushak, 1996)

Distance learning in North America tends to be delivered by universities. According to Bates (1995), these universities tend to be committed to the assumption that traditional "group, face-to-face instruction is the preferred and most effective form of higher education…and that the closer distance education can directly imitate this, the more effective distance education will be" (p. 167). Distance learning technologies tend to be chosen and used to deliver classes rather than to create flexible learning opportunities. For this reason, use of distance learning in the U.S. has been less flexible and innovative in serving the needs of students than it has been in Europe and Asia.

Cost Effectiveness of Distance Learning

Bates (1995) outlined general fixed and variable cost structures that can be expected using various forms of distance learning. Specific costs will vary based on specific situations. In general, interactive television tends to cost far more per student study hour than any other form of distance education. Both fixed production and equipment costs and variable transmission and labor costs tend to be high. Our own ITV classes demonstrated this; interactive television broadcasts to two distant sites required an instructor, two teaching assistants, three technicians, and three rooms full of sophisticated broadcast equipment. For the Internet courses, our students provided their own computers and Internet service, raising issues of access for those who may not be able to afford the equipment.

In summary, the evaluation of a distance learning project should focus on student learning, student attitudes toward the learning experience, faculty attitudes toward the experience, degree of institutionalization of distance learning delivery, the level of support required to implement distance learning, and cost effectiveness of the instruction relative to its goals.

EVALUATION QUESTIONS

The following questions were addressed in the evaluation components reported in this paper.

1. How satisfied are faculty, students, and project administrators with the distance learning experience?
1. Do students think they learn the class content effectively using the new technology? What factors enhance learning?

EVALUATION PROCESS

A survey of students and interviews with instructors, project administrators, and an ITV on-site coordinator were used to answer the evaluation questions.

Student Survey

Based on the literature and the research questions, the survey was constructed in three sections of close-ended questions: Content of Courses (7 questions), Delivery Method (11 questions plus an additional 6 questions for Internet users), and Administration (7 questions). The ratings provided a 5-point Likert scale of agreement ranging from "strongly disagree" to "strongly agree." A composite Overall Satisfaction scale was computed using the arithmetic mean of three "overall" questions from the previous three sections and 2 final satisfaction questions. An open-ended section provided respondents with the opportunity to provide unstructured feedback. The questionnaire was reviewed by two administrators from the program, a student who had been enrolled in distance learning, and two faculty members who had taught at least one of the classes, and revisions were made.

Survey Administration and Return Rate

All students (131) who were enrolled for one or more of the distance learning courses were mailed a questionnaire. Students who were part of an ITV class at the broadcast origination site (on the St. Paul campus) were not included in the survey though, in retrospect, they could have provided useful feedback. Two weeks after distributing the questionnaire, a telephone call was made to all non-respondents requesting their participation. Subjects with no telephone number on file or who requested another copy received a second questionnaire.

Seven surveys were returned as undeliverable, and 61 useable returns were received, for a return rate of 49%. This might have been increased if 1) it had been possible to make more than one reminder contact, 2) there had not been a Canadian mail strike, and 3) current addresses for all students had been on file. The Human Subjects Committee at the University of Minnesota would not approve more than one followup request. Of the 61 returns, 23 students had taken an ITV course, 26 had taken an Internet course, and 12 had taken at least one of each. Two entries were made in the data matrix for these 12 respondents, one for their rating of the ITV course(s), and one for their rating of the Internet course(s). Thus, the analyses reported here are based on course ratings rather than individual returns. The final n for this analysis was thus 35 ratings of ITV courses and 38 ratings of Internet courses (for a total n of 73).

Survey Reliability

Reliability was determined by calculating Cronbach’s alphas on the returned surveys. The reliabilities thus calculated, by section, were .88 for Content, .91 for Delivery Method, .68 for Administration, and .79 for Overall Satisfaction. These reliabilities range from acceptable to high (Rosenthal & Rosenow, 1991).

Interviews

Nine of the 14 instructors involved in the Project were interviewed by telephone. The remaining instructors were not available during the interview period. An unstructured interview guide was developed; the intent was to have the faculty share their feelings and observations about their experiences with the Project. Four administrators involved with the Distance Learning Project and one ITV on-site coordinator were also interviewed. Two of the researchers reviewed the transcripts to identify themes that emerged from the interviews.

STUDENT SURVEY RESULTS

The means of the four sections ranged from mildly (3.68) to strongly (4.18) positive. Respondents thought most highly about the content of the courses, followed closely by their overall opinion of the courses. A small gap appears between these ratings and the ratings of delivery method and administration. However, even these sections produced positive opinions. Table 1 shows the means and standard deviations for the four sections. Paired t-tests (70 to 72 df) revealed no difference in the means for delivery method and administration and in the means for content and overall satisfaction (p > .05). All other differences were statistically significant (p < .001) (Content vs. delivery method, content vs. administration, overall satisfaction vs. delivery method, and overall satisfaction vs. administration).

Table 1

Means and Standard Deviations for Four Survey Sections

Comparing ITV and Internet Classes

There were few differences in students’ ratings based on method of delivery (ITV vs. Internet). Though they saw little difference in content, method, and administration (all p’s > .1), respondents marginally preferred the Internet courses in their overall rating (M=4.26, SD = .98) to the ITV courses (M=3.89, SD = .86)(t (71)=1.75, p=.085). This marginal difference is largely the result of one item in the composite scale: "I would recommend to a friend that they take this course with this method." Respondents who had taken an Internet course (M=4.40, SD=1.13) were more likely to agree with this statement than were respondents who had taken an ITV course (M=3.42, SD=1.42)(t(69) = -3.21, p < .01). However, the means for both this item and the composite scale still cluster in the upper end of the rating scale. Again, this is an issue of a difference between moderate and strong praise for the program. There were no other significant differences in students’ opinions based on course delivery method (all p’s > .1).

In-depth Look at the Internet classes

Several questions were asked specifically of the participants in the Internet classes. Access to technology (both physical access and skill level) remains an issue for a substantial minority of the sample. Thirteen students (35.1%) rated themselves at or below the scale midpoint in terms of having adequate physical access to the needed technology for the course (M=3.68, SD=1.42). Respondents’ ratings of their skill level prior to the course also show a bimodal distribution, with 10 people (26.3%) below the midpoint of that scale (M=3.6, SD=1.41). However, in rating their technology skill levels after taking the course, only 3 students (8.1%) continued to mark themselves below the scale midpoint (M=4.3, SD=1.05). Thus the course provided an opportunity for some students to enhance their skills with the technology. Finally, all but one respondent marked at or above the scale midpoint for the statement, "I enjoyed using the technologies in this course." Thus, though many found the technology challenging, almost all were satisfied with their interaction with it by the time the course was over.

Dissatisfied Customers

There were 9 students whose scores on the 5-item "overall satisfaction" composite scale fell below the scale’s midpoint (one each of 1, 1.5, 2.0, 2.25, and five of 2.8). These were evenly split between the ITV courses and the Internet courses. However, 7 of these 9 were students who were not in the HRD program and 6 of the 9 were part time or occasional students. Perhaps being on the outside of the formal communication channels of the HRD program, being in a distant location from the university, and using an experimental program to take a class all conspired together to make their experience less than enjoyable.

Open-ended Responses

Most students taking ITV who made comments liked the experience overall. However, when they were critical, they most often focused on poor instructional methods used by some instructors, lack of accessibility to the instructor, the poor performance of some of the ITV technology, and the large size of the classes which limited participation with the overall group. They were most satisfied with the ability to access instruction not otherwise available, the effort by some of the instructors to make sure the course material came through clearly at all locations, and the use of discussion groups, high quality visuals and video resources, and outside speakers.

There were equally as many favorable as unfavorable comments about the Internet courses. The unfavorable comments centered around the inability to access course content because of difficulty with the technology, lack of access to the technology, lack of personal expertise with the technology, and the amount of work that was required. Those who responded favorably felt that the use of the Internet was appropriate for the subject matter, that the courses were well structured, and that the instructor kept in touch to give them feedback and make the course interesting and relevant. Favorable comments also cited the use of many of the communication capabilities of the Internet, such as chat-rooms and video conferencing, access to a number of exemplary sites and resources that were incorporated into the course, and frequent contact with the instructor and other students.

A common underlying theme in favorable comments for both media was the perception of significant amounts of "individualization" of the course. The reverse was true in the unfavorable comments; these students were unable to "connect" with the instructor, the course itself, or other students. Those who seemed pleased found a way to "take responsibility for (my) own learning and became comfortable with the technology."

Themes from Interviews with Instructors,
Administrators, and an On-Site Coordinator

A review of the transcripts of the interviews yielded the following themes.

1. Team Work, Cooperation, and Planning

Instructors reported that preparing for DL courses required far more time and attention to detail than preparing for traditional classroom courses. Team work and cooperation among instructors, technical media specialists, and administrators were important attributes of successful course preparation and implementation. Traditional classroom instruction, in contrast, tends to be a solo effort. For distance learning, course materials and course plans need to be completed well in advance of a specific class session so all team members can fulfill their particular responsibility.

2. Flexibility

On-the-spot flexibility is required when unplanned occurrences interfere with planned activities. The instructor is confronted with such issues when snow prevents students from reaching one of the sites, but students are present at the host site; when the system goes down for an hour in the middle of class; or when faxed materials are lost or the fax machine is down. In some respects, these situations are faced in a traditional class. Overhead projector bulbs burn out, some students are isolated in a snowstorm while others make it, and directions to support staff are misinterpreted. The magnitude of the issues, however, seems larger.

3. Using DL Technology

While offered to all instructors, few of those interviewed took advantage of specific training in using the DL technology. Difficulty with the technology ranged from "annoying" to "show stopping." For ITV courses technical difficulty sometimes caused portions, such as audio, to be missed, and even cancellation of entire class sessions. Noise, including paper rustling and electronic noise, was sometimes a noticeable distraction. This noise, along with forced protocols, such as waiting for a remote site to be recognized, was felt by some to detract seriously from the educational experience.

Technical difficulty in Internet courses was sometimes more subtle and cumulative in effect. According to reports from students to those interviewed, repeated difficulty in gaining computer access to the course(s) caused some students to drop out before completing the course while feeling upset with the technology and the University as an institution. Internet courses that pushed the "technology envelope" were felt to be difficult for many of the students because they did not have enough previous experience to make it work. However, courses that did not make good use of technology were considered boring. The courses cited as most satisfying to students were those that used several levels of technology (e.g., computer video conferencing, on-line delivery of content, reference to on-line resources, chat-rooms, and e-mail) to accommodate the technological ability and access level of the students.

4. Administration: Performance and Roles

The quality of service performed by the distance learning administrators was perceived to be widely variable. All past and current administrators interviewed expressed frustration with the lack of continuity between incumbents and the need to spend an inordinate amount of time finding out "how the system works" with respect to registration, tracking enrollments, disbursing funds, etc. It appears that clear operating procedures were difficult to establish within the existing University bureaucracy, difficult to document because distance learning required so many exceptions to established University procedures, and difficult to maintain because of high turnover in the DL administrator position, in one case due to an improved employment opportunity and in two cases due to performance deficiencies.

The Distance Learning administrators tended to occupy the position for one year. On two occasions the position has been split between two people. The duties of the position tend to cover a broad and ill-defined range that has included administration, coordination among the personnel and sites, technology troubleshooting, student communication and support, answering content questions on HRD, and instructional design. Few individuals who have occupied the administrator position felt that they did an adequate job of performing all of these functions, and many found the experience to be less than satisfying. The exception was when the position was shared among people with complimentary skill sets, and specific responsibilities were assigned to each person. Some also said that the expectations others held for their role were unreasonably high.

5. Student Educational, Communication, and Support Needs

The persons interviewed reported that some DL students enrolled in HRD courses simply because those courses were available. They saw these DL courses as their only option due to constraints in time or geographic distance. These same students tended to appreciate having access to higher education and were willing to overlook technological shortcomings in the delivery. Students interested in fields other than HRD often asked why other programs were not available to them via distance learning.

Some students in DL courses sought out the DL administrator or on-site coordinator because they felt that instructors were not easily available. Communication through a variety of means became more important since personal, informal classroom contact (before and after class, and during breaks) was rarely available. The communication methods used included telephone, individual conference via ITV before class, e-mail, and computer conferencing. Courses that used a broad array of communication were positively perceived to have built a "community." It was reported that ITV students, in general, felt that they could interact satisfactorily with the instructor