10.11 An Evaluation of Online Distance Education Course Databases

Editorial

You know, nothing makes me more cross than not being able to find information when I want it—especially when I am online. My favorite browser-based search engine is almost as good as having a well-trained research librarian welded to my upper left arm, but even that has its limitations. I am by no means a "newbie" at all this—and I would rank my computer skills above average, and my search strategies are well-honed—but if the database I am searching is poorly designed, and content is lacking, I get frustrated quickly and click to another site.

Many of those who are seeking distance education opportunities are new to the Internet and may not be especially skilled with computers. Some may not be native speakers of the English language. As the author of this month's article says,

An individual interested in taking a particular course, or in simply determining what is available, may be forced to locate, and then plow through, many unwieldy listings of online courses offered by individual institutions. The cumbersome process of locating the right course or program may discourage many from pursuing the distance learning option.

In the course of her evaluation of distance education databases at the behest of our colleagues at TeleEducation NB, Ms. Rose has developed a set of criteria for excellence in database content, process, and presentation of course information, and has focused on the usability of these databases for the prospective student.

I suggest that if you—whether you represent yourself individually or an institution—are planning, preparing, or revising your own public catalog of distance-delivered courses, take a long, hard look at these criteria and see how your own course database compares. For a prospective student, this catalog may be his/her first introduction to your offerings and your first opportunity to encourage that student to become your student.

Mauri Collins
DEOSNEWS Editor

An Evaluation of Online Distance Education Course Databases

Ellen Rose

INTRODUCTION

Progress in the area of accessibility and support for distance learners has not always kept pace with the growing distance education field. An individual interested in taking a particular course, or in simply determining what is available, may be forced to locate, and then plow through, many unwieldy listings of online courses offered by individual institutions. The cumbersome process of locating the right course or program may discourage many from pursuing the distance learning option.

In recent years, numerous databases of distance education courses have been developed and made available on the World Wide Web. These online databases are intended both to provide a central repository of up-to-date information on the many courses and programs that are available remotely and to make it easier for users to locate information about courses that meet their specific needs.

This article provides the results of an evaluation of twenty-one distance education course databases that are currently available online. Commissioned by TeleEducation NB, the evaluation used a standard set of criteria to assess each database in terms of its utility in the course-selection process, from the perspective of learners or potential learners having basic computer proficiency.

PROCEDURE

The first step in the database evaluation procedure was to develop a number of evaluation criteria in order to assess each distance education course directory according to consistent standards. The criteria were derived from a consideration of what a potential learnerand particularly a user with only basic computer skillswould need to facilitate his/her search for the right distance course and, ultimately, to register for the course. The criteria are described in the Evaluation Criteria section of this article.

The following twenty-five sites were identified for evaluation:

Following the development of standardized assessment criteria, each site was reviewed and evaluated in terms of how well it met each criterion. For each criterion, a rating was assigned as follows:

A The database meets the criterion satisfactorily. (This rating has a value of 1.)

+ The database meets the criterion in a way which exceeds the basic requirement. (This rating has a value of 2.)

– The database does not fulfill the criterion, or does so in a way which is not useful or functional. (This rating has a value of 0.)

N/A The criterion is not applicable to the database. For example, for a database that does not have search capabilities, the criterion “Search accuracy” is not applicable. (This rating has a value of 0.)

Each database was assigned a final Usability Rating, which represented the total value of the A, +, –, and N/A ratings that it received for the various criteria. (See the Findings section of this article).

Before the evaluation data was collected and subjected to analysis, a decision was made to eliminate four of the sites from the final analysis, since they do not truly qualify as databases of course information and would therefore skew the results. The sites that were evaluated but ultimately removed from final consideration are ABCentral, which simply provides links to Web pages matching search criteria; About.com, which provides a central repository of “expert” assistance on a variety of topics; CollegeNet, an online admissions system; and Name It and Frame It, an online book for people hoping to gain, via distance education, the credentials necessary to enter the Christian ministry.

EVALUATION CRITERIA

The following criteria are derived from a consideration of what potential learners would need to facilitate their search for the right course. A few of the criteria, such as “Information storing” and “Information sharing,” can be considered “nice to have” rather than essential. However, for the most part, the criteria are of equal relative importance, since the absence of any one of these features—whether “Ease of use,” “Keyword search,” or “Accessibility”—may tend to set up obstacles that could discourage users from pursuing the distance learning option.

The criteria have been divided into six categories: User-friendliness, Search Capabilities, Reliability, Course Offerings, Course Information, and Connectivity. A description of each of these categories and their criteria follows.

User-friendliness

Rationale: Since online distance education course databases are developed as tools for potential users, it should stand to reason that they will be developed with the needs and abilities of the user in mind. As with most software, however, the user's needs can become lost in the complexity of the project: ease of use is sacrificed to mere functionality, and clarity of display is lost in the welter of nifty programming features such as flashing, frames, and clever graphics. The criteria in this category are intended to evaluate whether or not the needs of the user—particularly the user who is relatively new to the online world—appear to have been kept uppermost in the minds of those designing the layout and functionality of the course directory.

Criteria:

U1 Ease of use. All options are clearly identified and easily accessed; users will usually have a clear sense of “where they are” in the directory and can navigate easily through it.

U2 Clarity of display. The screen is well laid out, uncluttered, free of distractions, and easy to read.

U3 Terminology. Information is presented using terminology that would be familiar to most users. Unfamiliar words (e.g., “accreditation” and “computer-mediated communications”) are defined if used.

U4 Online help. Online instructions for using the course database are available and are useful.

U5 Information storing. Users are able to bookmark or store course information (e.g., in a “shopping cart”) as they browse through the database.

U6 Distance learning information. The database offers easy access to general information on distance learning, studying online, diploma mills, etc. that will help those new to the world of distance learning understand what to expect and make their decision about an appropriate course and institution.

Search Capabilities

Rationale: No matter how extensive the distance education course database, it is ultimately only as good as its search engine. If potential learners cannot easily locate a course or program that meets their needs, then the database is not useful. The criteria in this category are intended to assess whether or not the user, even a “newbie,” would be able to locate a course on a particular topic with relative ease.

Criteria:

S1 Keyword search. The database allows users to perform a full text keyword search on a keyword or keywords of their choice.

S2 Advanced search. Users can narrow their search according to other criteria (such as course level, cost, etc.) or by entering a second keyword.

S3 Browse. The database allows users to browse through listings of the course offerings.

Reliability

Rationale: Like any piece of software, a distance education course database is useful only if it performs its intended functions smoothly and without error. The user may tolerate only so many URL “not found” errors, other programming glitches, and obvious inaccuracies before abandoning the search in frustration. The criteria in this category are intended to assess the robustness of the database from the user's perspective.

Criteria:

R1 System reliability. There are no system errors, crashes, or dead links.

R2 Information reliability. The course and program information contained in the database is up to date and appears to be correctly classified.

R3 Search accuracy. The keyword and advanced searches yield courses that match the search criteria.

Course Offerings

Rationale: The purpose of a database is to simplify the distance learner’s search by bringing together as many courses as possible into a single repository. If the database contains only a limited number or range of courses, its potential usefulness is seriously compromised. If it includes courses with limited distance availability—such as classroom-based courses with online components or online courses available only on-campus—then it is of little help for the user looking for courses that can be accessed and completed at home or work. And if the database contains only synchronous courses, delivered via such media as audio- and videoconferencing, then its usefulness to employed individuals hoping to upgrade their skills and knowledge may also be severely limited. The criteria in this category are intended to assess the relative usefulness to the distance learner of the courses contained within the database.

Criteria:

O1 Number. The database contains enough course offerings to be useful. In other words, for each category or topic, there is a variety of courses for the user to choose from.

O2 Range. The database contains courses on a broad range of topics.

O3 Institutions. The database contains courses from a variety of reputable institutions, not just from one or two locations.

O4 Online availability. The database contains a significant number of courses that are available completely online and are easily accessed from home or the workplace, with no residency requirements.

O5 Reach. The database contains courses from institutions around the world, and is not intended primarily to serve learners in a particular state, region, or country.

O6 Level. The database contains courses for a variety of levels, such as primary, high school, undergraduate, and graduate.

O7 Time independence. The database contains a number of asynchronous courses that do not start and finish on predetermined days and that can be accessed by users at any time of the day.

O8 Workplace training. The database contains courses that would be useful for employed individuals seeking opportunities for professional development and workplace training.

Course Information
Rationale: The database should provide complete course information upon which the user can base decisions about whether or not to enroll. If, after reading the course information, the user is likely to feel uncertain about what the course involves or whether or not he/she meets the prerequisites, then the database is not useful. The criteria in this category are intended to evaluate whether or not the database meets the user’s information needs with respect to individual courses.

Criteria:

I1 Standardized information. The database offers the same categories of information for all courses, rather than relying on each course provider to provide information as the provider sees fit. Having a standard basis of comparison makes it much easier for users to select appropriate courses. (Note that, for this criterion, a rating of + indicates that the database conforms to standard 1484.12 of the Instructional Management System (IMS)/Institute of Electrical and Electronics Engineers (IEEE), which is becoming the accepted standard for presenting data about educational institutions and programs. For more information, see <http://imsproject.com> and <http://ltsc.ieee.org/wg12>.)

I2 Medium. The database specifies the medium of delivery for each course.

I3 Type of learning. The database provides information on the pedagogy or teaching method for each course offering.

I4 Program. If applicable, the database provides information on the program the course is part of, the level of the course (e.g., graduate), and the number of credit hours the course is worth.

I5 Prerequisites. For each course, the database indicates any prerequisite skills or knowledges, or courses the learner must have already completed.
I6 Course objectives. The database provides information on the course objectives.

I7 Cost. The database provides information on the cost of the course.

I8 Target audience. The database provides information on the type of learners for whom the course is intended.

I9 Time. The database provides information on the course length, estimated time requirements, and deadlines for enrollment, if applicable.

I10 Language. The language(s) in which the course is delivered is specified.

I11 Residency. The database indicates whether or not there are residency and other onsite requirements (e.g., proctored exams) in order to complete the course.

I12 Technical requirements. The database provides information on the minimum hardware and software requirements for the course.

I13 Course provider. The database provides information on the institution delivering the course.

Connectivity
Rationale: A database that is made available on the World Wide Web is of most use to potential learners if it exploits the ability of the Web to connect users with course providers and each other. The criteria in this category assess the degree to which the databases transcend the limitations of stand-alone configurations (or, even worse, paper-based course catalogs), and the extent to which they facilitate the process whereby the user, having located the right course, can take the all-important next step: register for the course.

Criteria:

C1 Accessibility. The database does not simply provide course and contact information, but makes it easy for users to register for or access a course, or request more information, via hypertext links.

C2 Links to other databases. The database offers links to other online distance education databases.

C3 Conferencing. There is an easily accessed and easily used forum for users to ask questions or share information with other users about courses and institutions.

C4 Information sharing. There is a provision for the user to forward specific course or program information (e.g., to a boss or a human resources manager).

FINDINGS
The Distance Education Directories Comparison Chart (located at <http://teleeducation.nb.ca/media/reports.shtml>) contains a comprehensive chart showing the ratings assigned to all twenty-one online distance education databases, for all criteria. The final usability ratings of these databases are as follows:

Following is a summary of the key findings for each of the six evaluation criteria categories: User-friendliness, Search Capabilities, Reliability, Course Offerings, Course Information, and Connectivity.

User-friendliness
Most of the databases evaluated (seventeen, or 81%) rated satisfactorily (¯) for the ease-of-use criterion. Only one (4.8%) received a + rating for this criterion. Ratings for the clarity of display criterion were lower, with six (28.6%) of the databases receiving a poor (-) rating as a result of clutter or sloppiness.

The use of ill-defined terminology related to distance learning is a problem in five (23.8%) of the databases.

Some type of online instruction for using the database is available in only eight (38.1%) of the databases.

Only three (14.3%) of the databases offer the user an opportunity to store or mark information on particular courses as he/she pursues a search.

Fourteen (66.7%) of the databases provide easy access to information on distance education that would either facilitate the search or help a user decide if distance learning is practical. Of these, four (19%) provide extensive and very helpful information which merits a + rating.

Search Capabilities
While the ability to search the course listings would seem to be fundamentally important, not all of the databases provide this option. Sixteen (76.2%) of the databases allow users to perform a keyword search of the database contents. Of these, twelve (57.1%) allow users to refine the search with additional search criteria, such as mode of delivery.

Fifteen (71.4%) of the databases allow users to browse through course listings, and in two (9.5%), browsing is the only means of searching the database contents. Three (14.3%) of the databases have no search capabilities whatsoever.

Reliability
Given the difficulty of ensuring that all links are current and active, and that all information is up to date, it is perhaps not surprising that only fourteen (66.7%) of the databases rate satisfactorily in terms of system reliability, and that another fourteen (66.7%) rate satisfactorily in terms of information reliability. Only eleven of the sixteen databases offering keyword search capabilities rate satisfactorily in terms of search accuracy.

CourseOfferings
Four (19%) of the databases do not contain any course offerings, which explains the low cumulative scores for some of the course-offerings criteria.

Seven (33.3%) of the databases rate satisfactorily in terms of the number of course offerings, although not all databases specify the number of courses they contain—a fact that can be considered a deficiency in itself. In general, databases receiving a satisfactory rating for this criterion contain approximately 1,000 to 4,000 courses. The six databases (28.6%) receiving a + rating in this category are MindEdge, with about 9,000 offerings; CURSUS, with approximately 10,000 courses; TeleCampus, with 22,000; Hungry Minds, with 24,000; the International Centre for Distance Learning, with 31,000; and the Distance Learning Course Finder, with 50,000.

Fourteen (66.7%) of the databases rate satisfactorily in terms of the range of course offerings. Most (nineteen, or 90.5%) offer courses from a variety of reputable institutions.

It is often difficult to assess what percentage of a database’s course offerings are online; however, eight (38.1%) of the databases do contain some courses that are delivered via the World Wide Web. Only four (19%) of the databases merit a + rating in this category, suggesting that all of the courses that they contain are Web-based.

Since most of the databases consist entirely of US institutions, or are intended to serve only a particular region or province, most databases rate fairly low in terms of reach, with only five (23.8%) of the databases receiving a satisfactory rating and two (9.5%)—International Centre for Distance Learning and TeleCampus—receiving a + rating.

Most of the databases seem to be designed solely for people wishing to pursue a university education via distance and do not include primary, elementary, and high school upgrading opportunities. Therefore, only four (19%) of the databases received a satisfactory rating for the level criterion, while five (23.8%) received a + rating, indicating that they also contain courses and programs at the high school, and perhaps even primary and preschool, levels.

Thirteen (62%) of the databases offer a good selection of asynchronous courses and/or courses without fixed start dates, offering learners the opportunity to participate in courses at their own convenience.

Twelve (57.1%) of the databases contain a good selection of courses that would be appropriate and useful for working individuals seeking to upgrade their vocational skills.

Course Information
There is a very broad range in terms of the kind of course information provided by the twenty-one online distance learning databases. As noted above, four of the databases contain no course offerings and, therefore, provide no course information. Of the seventeen that do, most offer very minimal information, such as the course title and information on how to contact the institution delivering the course. Only two (9.5%) of the databases—CURSUS and TeleCampus—can be said to provide enough information (such as cost, objectives, prerequisites, time, and technical requirements) to allow users to make reasonable, informed course selections.

Thirteen (62%) of the databases provide standardized categories of information that allow users to make decisions based on a comparison standard, although, as noted above, the information provided is often very minimal. Only the TeleCampus database receives a + rating in this category, for meeting the IMS/IEEE standards for providing information on educational courses and programs.

Connectivity
A surprising number of the online databases fail to capitalize upon the capabilities of the Web. Fifteen (71.4%) of the databases do provide online links to the Web pages of course providers and e-mail contacts, but only ten (47.6%) allow users the opportunity to connect with each other in order to share comments about courses and institutions.

Only four (19%) of the databases help facilitate the user’s search by providing links to other online directories of distance courses. Another four (19%) of the databases have provisions that allow users to easily e-mail course information to a human resources manager, friend, or other person.

CONCLUSION
The findings of this evaluation suggest a lack of agreement among those institutions and organizations providing online distance learning directories regarding the role and functionality of such a database. Some databases offer search capabilities, while others do not; some provide course information, while others merely provide links to related Web pages; some function more or less as online catalogs, while others capitalize on the connectivity of the Web; and some are clearly not designed with the learner’s needs in mind, while others provide numerous options intended to simplify the user’s search for a course that meets his or her unique needs.

AUTHOR NOTES
Ellen Rose
Textpertise Writing Services
32 Estey Lane
Fredericton, NB E3A 5N2
Canada
Phone: 506-451-8883
E-mail: tws@nb.sympatico.ca
Ellen Rose is a lecturer at the University of New Brunswick and a consultant in the field of learning technologies. Her book, Hyper Texts: The Language and Culture of Educational Computing, is forthcoming in December 2000 from The Althouse Press.