10:4 The Effectiveness of Self-Tutorial Study vs. Classroom-Based Instruction for Real Estate Relicensure: An Experimental Study

Much research in distance education is conducted by academics in academic settings. This month’s article features a non-academic professional group attempting to determine the relative efficacy of their prescribed methods for fulfilling mandated professional development hours. This study illustrates the reality of researchers working within the design constraints of a contracted study, where the study design was dictated by the sponsoring agency. While the design was constrained, the research was carried out with painstaking thoroughness and reported with sufficient detail to encourage replication.

Mauri Collins
Editor

William E. Blank
Department of Leadership Development
University of South Florida

Kristine Y. Hogarty
Department of Educational Measurement and Research
University of South Florida

Jeffrey D. Kromrey
Department of Educational Measurement and Research
University of South Florida

James Witte
Adult Education Programs; 4096 Haley Center
Auburn University, AL 36849

This study compared print-based self-tutorial study with classroom-based instruction for completing mandated professional relicensure education for real estate salespersons and brokers in Florida. Pre and posttests were constructed based on a common textbook for both settings. A matched-pairs design was used in which the two highest scorers on the pretest were paired and randomly assigned to one of the two treatment groups, the next two highest scorers were randomly assigned and so forth. The classroom-based group completed their two-day, fourteen hour, continuing education course at one of two real estate schools. The self-tutorial group was sent a widely used "correspondence" textbook and given two-and-one-half weeks to complete their study. Both groups took the same posttest. Both groups posted a statistically significant gain in pre vs. posttest performance. The classroom group’s gain in performance was statistically significantly greater than the gain of the self-tutorial group. However, this represented only three additional questions answered correctly out of seventy-two (i.e., a 6.7 point increase for self-tutorial versus a 9.7 point increase for classroom).

Introduction

This study was commissioned by the Florida Real Estate Commission Education and Research Foundation Advisory Committee. Its goal was to provide empirical data to assist in resolving the ongoing controversy regarding the relative efficacy of self-tutorial study (most commonly referred to as "correspondence study" within the real estate profession) versus classroom-based study for completing the state mandated fourteen hours of continuing education for real estate salespersons and brokers in Florida. Much of the design and setting for the study was specified in the original request for proposals distributed by the Florida Real Estate Commission. The authors were awarded the project contract, however, the overall approach and particular research design that was specified may not have been the one the authors would have chosen had they conducted the study on their own.

Extent of Correspondence Study

The non-classroom instructional strategy that was the focus of this study was the correspondence format in which real estate professionals read and complete review questions in a book at their own pace in their own setting. While completing such a course, they may or may not "correspond" with a course instructor by mail or phone. Feedback from participants indicated that the majority of real estate professionals completing such correspondence courses rarely correspond with an instructor. In this study, participants who were randomly placed in the correspondence group, in effect, completed a self-tutorial program since there was no opportunity to interact with an instructor. Because, in practice, few people actually interact with an instructor, the correspondence scenario played out in this study closely matches actual practice. Since salespersons and brokers renewing their license have the option of contacting an instructor for clarification, additional information, or feedback, (thus making it truly correspondence), we began by looking at correspondence study historically.

Correspondence learning, home study, and other forms of distance learning have grown enormously in popularity during the past several decades. The forerunner of all distance learning formats in use today is correspondence learning or home study as it is often called. Michael Moore (1995) comments on the extent of correspondence education:

Learning in the comfort of one’s own home has advantages over learning in a context arbitrarily established by an educational institution. This is particularly true for working adults, such as real estate professionals, who have other significant demands on their time (e.g., job, home, and family). Correspondence learning is widespread in the United States and throughout the world both for university credit and in non-credit settings such as businesses and proprietary schools. It has proven to be a viable, effective alternative to the traditional classroom-based method of teaching and learning, particularly for the continuing professional education of working men and women.

Moore (1990) adds,

According to the U.S. Department of Defense, almost 700,000 members of the United States Armed Services participate in postsecondary correspondence courses each year. In addition to the four million or so students involved in correspondence study through National Home Study Council (NHSC) institutions, over 300,000 people study each year by correspondence at the secondary and postsecondary levels through colleges and universities affiliated with the National University Continuing Education Association. Most of these students are studying for college credit.

More than ninety percent of distance education enrollment in the United States occurs through postsecondary institutions other than colleges or universities. About four million proprietary students are enrolled at any given time in schools that the NHSC accredits. International Correspondence Schools, which is both the first and the largest home study proprietary school, enrolled 70,000 to 100,000 persons a year during the 1980s and has enrolled over nine million students since 1890 (Verduin and Clark 1991).

Distance education, particularly correspondence study, plays a significant role in the upgrading, retraining and continuing professional development of millions of workers in the workforce. Speaking of the importance of correspondence education for retraining workers, Walsh (1971) related:

The purpose of this study was to determine the relative effectiveness of self-tutorial instruction versus in-class instruction for real estate relicensure in the State of Florida.

Method

Research Design. This study used a pretest/posttest control group design that incorporated the matching of participants prior to random assignment to one of the two treatment conditions for completing the continuing education requirement. A major strength of this design was to insure that the participants’ varying levels of pretreatment knowledge was evenly distributed between the two treatment groups, thus eliminating the possibility of placing more participants into one group who already possessed a higher (or lower) level of knowledge of the intended content. This was accomplished by first pretesting all the participants, then pairing the two highest scoring participants, and randomly assigning one to the self-tutorial group and the other to the classroom group. The next two highest scoring participants were then randomly assigned to one of the treatment groups, and so on. In addition to matching participants on pretest scores, the researchers also matched pairs as closely as possible by gender, income level, ethnicity/race, realtor and broker status, and geographic region.

Participants. The study involved both real estate salespersons and brokers. Requirements for the selection of study participants included:

1. The sample was to be equally divided between each of two major metropolitan areas in the state.

2. All participants were required to have at least five years, but not more than ten years, of real estate experience.

3. All participants had to be full-time real estate practitioners who held current active Florida real estate licenses, which were up for renewal.

4. All participants were required to have an income level of at least $30,000 gross annual salary from real estate activity.

Sample Size. A target sample size of seventy participants was sought. This sample size was determined on the basis of a statistical power analysis of the proposed design (Cohen 1988). Such an analysis is used to estimate the probability of rejecting a statistical null hypothesis (i.e., correctly determining that treatment effects are actually present) based on four parameters of the research design: (1) the Type I error rate; (2) the sample size; (3) the actual magnitude of the treatment effect; and (4) the type of statistical analysis conducted. The power analysis suggests that the sample size with repeated-measures analysis of variance, a Type I error rate of a = .05, and with a true difference in the effectiveness of the two treatments, as measured by the standardized mean difference, or effect size of 0.5 (i.e., a medium effect size), provides a probability of rejecting the null hypothesis of no difference in effectiveness of 0.82 if the correlation between the pretest and posttest is 0.50.

A medium effect size was used in these computations because this is the smallest effect that is probably of interest to the Florida Real Estate Commission Education and Research Foundation Advisory Committee. The standardized mean difference of 0.5 represents a one-half standard deviation difference between treatment means. That is, if both groups of subjects begin at the same knowledge level on the pretest, their posttest performance is expected to be one-half a standard deviation apart. Cohen (1988) describes this size of an effect as the typical effect size seen in published research in the social sciences. If the actual magnitude of the treatment effect is larger than 0.5, then the sample size will provide even greater statistical power.

Selection of Study Participants. Securing a sufficient number of study participants who met the specified criteria proved to be a major challenge in conducting the study. The project team purchased mailing labels of potential study participants from a private testing firm in the state. The researchers estimated that distributing 2,000 letters of invitation would be sufficient to yield 70 study participants. Mailing labels were drawn from the two targeted metropolitan areas by including approximately every fourth or fifth name in the database of those whose license was up for renewal during the coming year. These labels became the basis for soliciting study participants.

Letters of invitation and a participant data sheet were mailed to the salespersons and brokers selected by this sampling procedure. The letter indicated that participants would be offered a stipend of $250 for their participation. Sixty-four individuals returned the data sheet and agreed to participate in the study. In an effort to increase that number, an additional 200 mailing labels were obtained and a second mailing was completed. This yielded an additional eight participants for a total of seventy-two individuals who volunteered for the study.

A fairly large number of participants began dropping out of the study due to conflicts with the scheduled pretesting, previously scheduled vacations, real estate closings, related conflicts with the dates of the scheduled classroom instruction, or for other reasons. Additional recruitment efforts using a broadcast fax from the local association of realtors to the real estate offices in the two metropolitan areas yielded an additional twenty-one volunteers. Of the ninety-one participants who agreed to participate in the study, twenty-three matched pairs (forty-six participants) completed all aspects of the study.

Discussions were held with the funding agency and with owners of several real estate schools and suppliers of textbooks. It was determined that a common textbook was currently in use in Florida for both classroom-based and correspondence courses. This textbook was selected for use by both treatment groups in the study to control for the extraneous variable of course textbook. The textbook selected is regarded by real estate subject matter experts as a well-designed and comprehensive book to use for relicensure. Selection of a textbook that was in wide use for both classroom-based and correspondence study added a degree of authenticity to the study.

Test Development. Subject matter experts (SMEs), who were experienced real estate salespersons and brokers, were identified and invited to participate in a test item development workshop in which an in-depth overview of multiple choice test item writing principles and sample items were provided. Actual copies of the textbook, an outline of objectives from the textbook, and detailed instructions in writing test items for each objective were provided. The researchers were available throughout the item writing workshop to provide technical assistance, clarification, and to react to draft items. Approximately 130 test items were generated during the workshop (providing 2—3 items for each objective). The researchers reviewed the items, made editorial changes, verified the item content classification, eliminated a few poorly constructed items, and randomized the order of correct responses.

Next, pilot testing of the test items was conducted to determine their psychometric soundness. Pilot test participants were sought who had similar characteristics as the subjects who would be participating in the full study. A broadcast fax was sent to real estate offices in one of the target areas seeking volunteers to participate in the pilot testing of the pretest/posttest test items. An incentive of $50 was offered for the pilot testing session. Forty salespersons/brokers participated in the pilot testing sessions.

The analysis of the pilot test focused on item difficulty, item discrimination, and distractor performance. With regard to item difficulty, the ideal test item is neither too easy (too many people get it correct) nor too difficult (too many people get it wrong). Items with difficulty indices of less than 0.25 were flagged for special attention. The item discrimination index is an indication of how well each individual test item discriminates between those getting high scores and those getting low scores on the remaining items of the pilot test. The objective was to have test items answered correctly by those whose overall pilot test scores were high and vice versa. Items with negative discrimination indices were identified as being potentially confusing to more knowledgeable examinees. Finally, distractor analyses were performed to be certain the incorrect options were perceived as plausible by some examinees.

The next step was to have two additional SMEs, who were not involved in the initial development of the test items, review the results of the pilot testing. A test validation session was conducted involving two experienced real estate salespersons who also had extensive teaching experience. A printout with data on the results of each test item based on the pilot testing was provided to the validators with instructions for interpreting the pilot data. The researchers reviewed the completed work of the test item validation session and made any necessary editorial changes to the test items.

After the test items were refined using input from the test validation team and technical and editorial modifications by the researchers, two parallel forms of the final test were generated: Form A, to serve as the pretest, and Form B, to serve as the posttest. Because several test items were generated for each objective covered in the textbook, multiple items written for the same objective were randomly assigned to either the pretest or the posttest. The resulting tests each contained seventy-two items, with approximately fifty percent of the items common to both forms (to serve as anchor items).

Treatment Protocol. Pretest answer sheets were machine scored and the participants were paired by their number of correct responses. The top two scoring participants were paired and were then randomly assigned to either the self-tutorial group or to the classroom group. This randomization was done using a coin toss. Where possible, the researchers tried to balance the two treatment groups by gender, location, and their realtor status as well as on pretest scores that were within two points of each other.

After participants were assigned to one of the two treatment groups, a notification was faxed indicating to which of the two treatment groups they had been assigned. Unfortunately, at this point a number of participants withdrew from the study. Some indicated that they really could not or were not interested in attending a formal class and had originally volunteered to participate in the study based on the fifty-fifty chance they would be placed in the correspondence group. Others who were assigned to the classroom group indicated they had closings, vacations, or other conflicts with the dates of the classroom instruction. What made this attrition troubling was that when a participant dropped out, his or her matched pair was effectively eliminated from the study as well. Other participants, on the other hand, displayed unusual flexibility by driving to the classroom session in another part of the state if their schedule conflicted with the classroom session scheduled in their own area.

Two well-regarded real estate schools delivered the classroom-based instruction. One instructor from each school was selected and cautioned to conduct the two-day continuing education class just as they normally would. However, they were instructed to save one hour at the end of the second day to administer the posttest.

Participants placed in the self-tutorial group were sent notification and a copy of the textbook with instructions to complete the self-study materials as they normally would if they were taking a correspondence course for relicensure. They were given approximately two-and-one-half weeks to complete the self-tutorial course. The reader is reminded that study participants in the self-tutorial group had no contact with an instructor while they were completing the course on their own. This was consistent with the design called for by the contractor and may or may not accurately reflect the situation with actual real estate professionals who complete correspondence study.

Participants in the classroom group took the posttest at the conclusion of the second day of the class, which was proctored by the course instructor. Participants assigned to the self-tutorial group were sent a fax indicating where they were to attend a posttest session. The posttest for the self-tutorial group was proctored by one of the researchers. A total of fifty-two participants completed all aspects of the study and were paid the stipend, however, the data from only forty-two participants (twenty-three matched pairs) could be used in the final analysis because individual members of several matched pairs had dropped out of the study.

Results

The data indicated that the performance of both groups increased on the posttest relative to the pretest. The self-tutorial group’s pretest mean was 44.65 (SD = 6.78) and their posttest mean was 51.35 (SD = 7.15), for a gain of 6.7 points. Similarly, the classroom group evidenced a pretest mean of 43.17 (SD = 6.37) and a posttest mean of 52.87 (SD = 4.70), for a gain of 9.7 points.

The statistical significance of the results was evaluated through an analysis of variance (ANOVA). The results of the ANOVA follow. Because of the matching of participants in the two groups, both factors in the design (treatment condition and time of test) were within-subjects factors. The Instruction main effect tested the difference in overall mean scores for the two groups (i.e., averaged across both pretest and posttest). No significant differences were evident (F (1,22) = 0.00, p > .05). Similarly, the Time main effect tested the difference in overall mean scores for the two occasions of testing (averaged across treatment groups). This difference was statistically significant (F (1,22) = 60.28, p < .05), indicating that the participants as a whole increased their test performance significantly from pretest to posttest. Finally, the Instruction X Time interaction tested the difference in change from pretest to posttest for the two groups. This effect was also statistically significant (F (1,22) = 4.93, p < .05), indicating that the classroom instruction group demonstrated statistically significantly greater gains from pretest to posttest than did the self-tutorial group.

In addition to hypothesis testing, the data were analyzed by calculating the effect sizes for the two groups. The effect sizes were calculated by dividing each group’s change in mean scores (from pretest to posttest) by the group’s pretest standard deviation: Effect Size (d) = (Mean_post — Mean_pre) / SD_pre. This effect size indicates the magnitude of change in scores, in standard deviation units. The resulting effect size for the classroom instruction group (d = 1.52) was 50% larger than the effect size for the self-tutorial group (d = 0.99). The self-tutorial group’s posttest performance was nearly one standard deviation higher than their pretest performance (a very large change). However, the classroom instruction group’s posttest performance was more than one and one-half standard deviations higher than their pretest performance.

This study was commissioned to shed light on the debate regarding the relative effectiveness of correspondence study versus classroom-based instruction for delivering the mandated fourteen hours of continuing education for real estate salespersons and brokers in Florida. Although a substantial amount of research has been reported in the literature regarding the efficacy of correspondence and related non-traditional means of learning, very few controlled, methodologically sound studies have been completed. Virtually no previous research in the real estate field could be found. Although the non-traditional instructional approach of interest in this study is commonly referred to as "correspondence study," participants in this study randomly assigned to this treatment group did not have access to an instructor–not unlike the way actual practice often plays out within the real estate profession.

One of the primary purposes of requiring real estate and other professionals to participate in periodic continuing professional education is to protect the public interest by assisting practitioners to keep up-to-date with changes in their profession. Both self-tutorial and classroom-based study methods appear to be effective in this regard.

Of interest was the finding that the classroom group’s gain in performance was statistically greater than the gain of the self-tutorial group. The reader should be cautioned not to rush to the conclusion that classroom instruction is necessarily superior to self-tutorial instruction. Although the evidence suggests that the classroom group gained statistically significantly more information in their class than did their self-tutorial counterparts, their additional increase in performance was only three additional correctly answered questions out of seventy-two (i.e., a 6.7 point increase for self-tutorial versus a 9.7 point increase for classroom). While this difference in the performance of the two groups is statistically significant, the reader must put these technical results into the context of professional practice. Whether or not this "statistical edge" translates into the classroom group possessing a higher level of knowledge actually applied in the real estate field, or committing fewer mistakes, or having fewer complaints lodged against them, must await further study.

There are additional factors that the reader should keep in mind when interpreting these findings. Several factors in this study that would limit its generalizability for the entire state can be stated as follows:

1. It was conducted in only two regions of one state.
2. The study was conducted in metropolitan settings. How similar are these settings to more rural settings throughout the state?
3. The study was conducted with one textbook. Could the use of a different textbook have yielded different results? Was the textbook selected, perhaps more suited to classroom use than correspondence or vice versa?
4. The correspondence participants received no ongoing support from the researchers or an instructor once they were sent the correspondence materials. How representative is this of correspondence courses currently offered? Do salespersons who complete correspondence courses seek feedback from the correspondence school via progress checks, phoned-in questions, and so forth?
5. The self-tutorial group had approximately two-and-one-half weeks to complete the course. Was this a shorter or longer time period than most people typically spend on a correspondence relicensure course?
6. A considerable number of participants originally volunteering to participate dropped out of the study. Did this alter the representativeness of the group of participants remaining? Could it be that most of those who dropped out were more motivated or self-directed than those who remained?

Given these limitations, however, several conclusions appear to be appropriate, based on the data from this study.

1. Both self-tutorial and classroom-based instruction appear to be effective means of continuing education for real estate professionals. Both groups significantly improved posttest performance compared to pretest performance.
2. Real estate salespersons and brokers currently practicing seem to already possess a considerable amount of the knowledge covered in the textbook used in this study. They correctly answered approximately sixty-three percent of the pretest questions before they were exposed to one of the two treatment conditions.
3. The classroom-based approach yielded statistically significantly greater gains than did the self-tutorial method for the participants in this study, for the two instructors involved in delivering the classroom-based instruction, and for the textbook selected for use.

It is recommended that a series of follow-up studies be conducted during the next few years to build on the findings of this study. Only when several solidly designed studies are conducted, and those findings are viewed collectively, will it be possible to make evaluative judgements regarding the relative effectiveness of correspondence versus classroom instruction for providing the mandated continuing professional education for real estate salespersons and brokers.

William Blank is Professor in the Department of Leadership Development; Jeffrey Kromrey is Professor and Kristine Hogarty is a Doctoral Candidate, both in the Department of Educational Measurement and Research, all at the University of South Florida in Tampa. James Witte is Assistant Professor in Adult Education Programs at Auburn University.

Dr. William Blank
University of South Florida; Bldg. EDU 162; Tampa, FL 33620
(813) 974-0314; E-mail: blank@tempest.coedu.usf.edu

Top of Page