10.9 Evaluating Students' Online Course Experiences: The Virtual Focus Group

Editorial

I can remember the end of each semester in my days as a graduate student: taking fat envelopes of student evaluations to classrooms, standing and waiting while students dutifully blacked in the little bubbles on their sheets, collecting the sheets, stuffing them back into their envelopes, and dutifully trotting them back to the department secretary. I'd generally give a few minutes of derisive thought to the "smiley sheets" that gave students an opportunity, with twenty-five set questions, to express their satisfaction with the course, faculty member, and delivery technology, all bundled together.

The authors of the following article have been testing an innovative blend–a focus group and a modified Delphi methodology–to verify the important matters that made an impact on students' satisfaction with their online course(s). A focus group is an information-gathering technique that brings people together to respond, as a group, to questions posed by the group facilitator. These are usually "one-off" events, limited to a one- or two-hour discussion. The Delphi technique has the participants making their initial statements in writing; then, sometime later, they read each other's statements and have an opportunity to modify their own statements in light of what others have written.

This combination technique, taking place online at the students' convenience, gathers considerably more useful information for the evaluator and allows exploration, in depth, of all aspects of the course that contribute to student satisfaction. It provides richer feedback for the faculty member and course designer.

"Feedback," in all its manifestations in technology-mediated online courses, is quickly rising to the top of the tangle of Gordian knots that need to be cut. This study makes a good start.

Mauri Collins
DEOSNEWS editor

Evaluating Students' Online Course Experiences: The Virtual Focus Group

Catherine Schifter and Dominique Monolescu

Abstract

Students taking courses through the Internet want to be in control of when they access their courses and to choose when and how they interact with their classmates and instructors. In other words, students want flexibility. Because online courses require different preparation, infrastructure, technical support, technology expertise, and course delivery, evaluation of student online course experience(s) also requires a different evaluation paradigm.

The purpose of this paper is to present an alternative student satisfaction evaluation method to the traditional Likert-scale survey, and to demonstrate the quality and extent of the information that could be obtained. In this study, focus groups were combined with the Delphi technique to verify important issues that impact student satisfaction with online courses. This study is unique because the authors adopted asynchronous communication into the focus group research technique. This paper suggests that a virtual focus group (VFG) using a modified Delphi technique can be used to evaluate students' online course experiences.

Introduction

Distance education courses are becoming important options, not only for students who cannot easily commute to a university campus, but also for students who attend on-campus classes. Distance is no longer the primary reason for taking an online course; flexibility of time and access is of significant importance.

Distance education has matured significantly since the advent of the World Wide Web. Courses are much more dynamic and interactive, not only through discussion, both synchronous and asynchronous, but also with inclusion of audio and video technologies. Online course management systems (WebCT, Blackboard, and others) provide many features for enhancing a course that had not been easily available to the typical faculty member.

Given that most college students, like most college faculty, are new to online distance education courses, are they satisfied with their experiences? What worked best for them, and what suggestions would they have to make to improve the experience? Given that the online course environment is quite different from that of the classroom, should a different assessment model be considered to determine student satisfaction?

Toward the end of many traditional courses, a student evaluation survey is distributed. Because students are not experts in the subject matter, they can only attest to the course delivery, their involvement in the course, and their overall level of satisfaction. Surveys present predetermined questions that relate to the traditional course environment, and often use a Likert-scale approach (i.e., collection of quantitative data). Are the same questions applicable to an online course environment? Could another method, such as an online or virtual focus group (VFG), using a modified Delphi technique, elicit information more specific to the nature of the online course?

This paper suggests using VFGs as an alternative approach to evaluating student online course experiences through a highly structured opportunity for students to interact online, reflect on their online course challenges, and define together the important issues about online course environments that lead to or away from their course satisfaction, which can translate into student retention.

Focus Groups

Focus groups, used in marketing research since the 1940s, are designed as a real-time variation of the group interview. Groups vary in size from six to ten people who are not familiar with each other ( Krueger 1994). The participants are selected based on specific characteristics (that are relevant to the study), such as a previous experiences with a product or service. The focus group allows participants to discuss a product or service; to hear what others say; and to agree, disagree, or expand on each other’s ideas. Krueger (1994) notes that focus group methods allow participants to discuss each of the issues from their own viewpoints. The fact that people "listen to others’ opinions and understanding" helps them clarify their own (Marshall and Rossman 1995, 84).

Focus groups require a good balance between the participation of the moderator and the participants. Too much moderator control reduces the productivity, while too little control results in wasted time (McDonald 1993). How are focus groups different from group interviewing? A focus group relies on participants’ reactions to topics and to questions by the moderator. In contrast, a group interview involves an assembly of people, all exposed to the same questions. The main purpose of focus groups is to promote group discussion, while group interviewing is centered more on "the questions and responses between the researcher and participant" (Gibbs 1997, 1).

Virtual Focus Groups (VFGs)

Some marketing firms (e.g. Greenfield Online, Harris Black International, and NPO Interactive) conduct VFGs by having participants from different locations connected to their server simultaneously (Maddox 1998). Most of these VFGs are conducted in chat rooms, in real time.

The authors' project, however, did not require participants to be synchronously online. The researchers consider the technique described here to constitute a VFG because the process followed the six components that guide the focus group process: question formulation, participant selection, the use of a protocol, conducting the focus group, data analysis, and reporting the findings (Brotherson 1994).

Methods

In the summer of 1998, seventy-two students who participated in one or more online courses received an e-mail invitation to participate in this VFG. Because of a concern in preserving anonymity, the researchers used a special editing command on the university's e-mail system. Students were not able to identify which other students received an invitation. All contact between the researchers and the students was done via e-mail. Because student anonymity was preserved, it was not possible to identify the particular course taken or any participant demographics (e.g., gender, age, or status toward degree).

Eight students (11 percent of the original population) accepted the invitation to participate in the VFG. Over a period of fifteen days, they received a second e-mail message with more information and instructions regarding the general time frame for the VFG; the importance of maintaining anonymity by using a pseudonym for all postings; the researchers' expectations about the students' participation in the VFG; the number of students participating in the VFG; the days to be used for reading and the days to be used for writing impressions (to ensure all students had time to read and write about the other students' statements); and where to receive their token reward for participation. The students were asked to read posted questions and comments and to respond to them. They also were asked to read each other’s comments and to write their impressions about the statements made by the other participants. Students were given the opportunity to participate in the VFG at their own convenience by logging on to a URL set up for this study.

Specific questions were initially posted for student response. The focus group method was then combined with the Delphi technique, a systematic communication protocol, using experts (in this case the students from the courses) for forecasting (Linstone and Turoff 1975). Students were informed that, in the second week of the project, they were expected to read each other’s comments and respond with their own comments. All comments were shared with all participants, as would be done by mail, and much more slowly in a typical Delphi study. Students submitted nineteen responses to the seven general questions posted by the researchers, and made twenty-four comments after reading the messages posted by the other members of the project.

This VFG had three phases. During a two-week period, students had five days to post their initial comments to topics that were adapted from a previously used online, open-ended survey of student satisfaction with online courses. The questions examined (1) technology and content, (2) interaction with classmates, (3) interaction with professors, (4) expectations, time, and assignments, (5) advantages, (6) disadvantages, and (7) recommendations. After phase one of the VFG, students had a period of five days to read all the other posted comments. Finally, for the last phase, students posted their final thoughts about each of the seven topics, taking into consideration all the postings made by the other focus group participants. All students' postings were entered into a database and sorted by topic, date (prior or after phase two), subject, and nature of the comment prior to analysis.

Results

Eight students participated in the VFG interaction via the URL set up for this study. They all followed the instructions, and posted messages at three different times and on three different days. These students were very responsive, given their flexibility for participation.

Regarding technology and content of online courses, student 4 (S4) said that it is important "that the content of the course be well-suited for cyberspace." Examples of other comments follow

• For those of us who have to work full-time and attend classes, the ability to work at your own pace and log on at odd hours is a real advantage. (S1)

• In all of the online classes I have had, the work load was manageable. I was a little more disciplined knowing that I had to log on regularly to keep up with assignments, comments, and postings rather than waiting to discuss everything in a class setting once a week. (S1)

• I liked not having to be somewhere at a specific time. It is easier to schedule school around my family and work rather than work and family around school. The flexibility was wonderful. (S5)

• …I worked harder at this one, and learned more because of that. For this class I had to read the assignments and do the work. For a class at the school you can fudge the reading because you know you can hear the lecture. (S5)

Many students today must support their education by working part-time or full-time jobs. Relevance of content and flexibility in accessing the course, coupled with motivation to do the work on their own time, made the online course option viable. These were "experienced" students. The information would not be obtained through the typical student satisfaction survey without specifically asking for it. Will college freshmen be successful in this environment? What will be the impact of online courses for high school students?  Only time will tell.

Regarding "interaction in their courses," students said that, because of the nature of the content, some courses provided more opportunities for interaction than others. Nevertheless, S2 wrote that "eye contact, body language, and the human voice" are important in the learning process for him/her. Some students, with prior experience only in traditional courses, may have come to depend upon the nonverbal, synchronous communication with which they are familiar, but that are not available through online courses. S2 recommended the presence of dynamic classroom participation for online courses, again referring to the language used to describe the traditional course environment. On the other hand, S7 wrote

Interaction, much like beauty, is in the eye or expectations of the beholder. What is satisfactory for one may not be for another.

The concept of video interaction in the online course was introduced in one question. S1 wrote that it could be "slightly less effective," because it could be taking away the sense of anonymity. Interestingly, a face-to-face class meeting was suggested to deal with administrative matters such as explaining papers, making schedules, and demonstrating the computer technology to be used in the online course, such as an online discussion list, sending an assignment as an attachment to an e-mail message, and other aspects, depending on the online course management system being used. Overall, students perceived their online classroom interaction as making them feel "more included". S5 felt that the level of interaction was " probably more than normal." He/she felt more included in his/her online class, and stated "I feel rather alienated when going to class. With the online class I did not."

The design of the course, the manner in which the professor conducts/teaches/facilitates the course, and the content were significant issues for online course satisfaction for these students. Some students need more visual cues than others, so the need for visual interactions differed among the participants. While the anonymity of the online environment enhanced interaction and postings for some students, personal impressions were difficult, or impossible, to develop.

How important are these issues to overall course satisfaction? Education has been described as a social activity; therefore, if the social component is difficult to identify, how is the educational experience changed? While traditional student satisfaction surveys ask questions to determine how a professor conducts a course and whether the content is significant to the students, these concepts are perceived differently in an online course. It is important to understand the "experience" of online students in order to appropriately support them, and this information comes forth through the VFG.

Regarding the amount of time spent on assignments for face-to-face versus online courses, students wrote that, when taking an online course, they could use what had been their travel time to do class work. Some students proposed that, in the online class, they spent more study time than they would in a face-to-face class. When comparing, S2 wrote that it "overall equaled out." Students said they enjoyed the structure of having to meet deadlines for their online assignments. According to S5, he/she was forced "to be more diligent."

These students stated that the number-one advantage stated by these students for the online course experience was flexibility. In some cases, anonymity was highly appreciated, especially when dealing with a sensitive topic such as race and racism. On the other hand, not knowing how to deal with the technology can create frustration and be a disadvantage to the online classroom experience.

Student Recommendations

Students stated that professors’ ability to integrate their course work into the online environment was very important. Simply posting notes or materials to the course site was not enough for these online students. They wanted the interactivity and assignments that used the resources of the Internet. The students suggested interacting with instructors prior to the beginning of the course to understand the expectations and to understand how the course would be conducted. They stated that, to succeed in a virtual-learning environment, a student should be self-disciplined and, therefore, freshman students might not be ready for online courses. However, first-year college students who took online courses in high school may be fully ready for college online courses, and, in fact, may have higher expectations than students who did not take online courses in high school. Again, only time will tell, particularly since relatively few online high school courses are available to date.

Most of the respondents were not "distant" students, but were taking online courses for flexibility in their schedules. Although many students encouraged the institution to offer more courses online, they still valued the face-to-face environments. S1 stated that both students and administration should be committed to this learning process and suggested that technology problems must be solved as soon as possible to reduce frustrations (especially in summer courses "where there is no time to lose").

Based on these student comments, the key issues to success in an online course include student motivation to complete the course based on the quality of the course presentation, the interaction between students and the professors, and having clear course schedules and assignments. Could these concepts be addressed in a traditional Likert-scale survey? Yes; however, the richness of the students' comments would be missed in the attempt to quantify responses. Do student characteristics (e.g., age and gender) make a difference in perceptions? This paper cannot address the impact of student characteristics on satisfaction because, to maintain the anonymity of the participants, these characteristics were not collected.

Conclusion

The findings of this study are not earth shattering. For years, the literature on distance education has indicated for years that students need to be self-motivated and technically savvy to be successful in an online course. Interaction, both between students and students and between students and faculty, has been considered essential.

The uniqueness of this study is in adopting asynchronous communication into the focus group research protocol to assess students' satisfaction with their online course experiences. Given the importance of e-mail and Internet resources in courses increasingly offered by universities and colleges, this pilot study gives reason to suggest that VFGs can be a useful tool to evaluate students' course experiences. The researchers suggest that the major advantages of VFGs for students are their flexibility in both time and location. Students can give significantly more meaningful qualitative input, rather than merely checking numbers from a Likert-scale survey.

So far, focus groups have been considered a synchronous (real time) communication mode whereby people must come together in a physical space to express their opinions and feelings about a particular topic. By incorporating an asynchronous communication mode to the focus group technique, the researchers suggest that a richer data set can be obtained than is possible through the traditional survey method.

Limitations of the Study

This study was a single case study at one urban university, and participants were taking online, summer session courses from only one college within that university. External validity of the data gathered is extremely limited, since there was a limited sample size and all participants were volunteers, with their own motives for participating.

Future Research

Future research is needed to demonstrate whether VFGs are applicable to other settings, and whether the data they provide are comparable to the data from other online learning evaluation processes.

References

Brotherson, M. J. 1994. Interactive focus group interviewing: A qualitative research method in early intervention. Topics in Early Childhood Special Education 14: 101-18.

Gibbs, A. 1997. Focus groups. Social Research Update [online], Winter. Available online at http://www.soc.surrey.ac.uk/sru/SRU19.html

Krueger, R. A. 1994. Focus group: a practical guide for applied research. Newbury Park, CA: Sage Publications.

Linstone, H., and M. Turoff. 1975. Delphi method. Reading, MA: Addison-Wesley.

Maddox, K. 1998. Virtual panels. Advertising Age 69: 34.

Marshall, C., and G. B. Rossman. 1995. Designing qualitative research. Thousand Oaks, CA: Sage Publications.

McDonald, W. J. 1993. Focus group research dynamics and reporting: An examination of research objectives and moderator influence. Journal of the Academy of Marketing Science 21: 161-8.

Author Notes

Catherine Schifter, Ph.D.
Associate Professor, Carnegie Scholar, and Director of Academic Technology
Temple University
1301 Cecil B. Moore Ave.
667 Ritter Annex Bldg.
Philadelphia, PA 19122

Phone 215-204-3477
E-mail schifter@nimbus.temple.edu

Dominique Monolescu, A.B.D.
OnLine Learning Program Interim Director
Temple University
1486 Yost Road
Blue Bell, PA 19422

E-mail dominiq@astro.temple.edu

The American Center for the Study of Distance Education (ACSDE)
The Pennsylvania State University
College of Education
110 Rackley Building
University Park, PA 16802-3202
Phone (814) 863-3764  FAX (814) 865-5878
ACSDE@psu.edu
www.ed.psu.edu/ACSDE

©2001 The Pennsylvania State University
College of Education