ASSESSING THE VALUE OF WORKFORCE TRAINING
A GUIDE FOR SMALL AND MID-SIZED COMPANIES AND THE PROVIDERS
THAT SERVE THEM
NATIONAL ALLIANCE OF BUSINESS
ASSESSING THE VALUE OF WORKFORCE TRAINING
By Eunice N. Askov
Institute for the Study of Adult Literacy
The Pennsylvania State University
John Hoops
Western Massachusetts Chapter
National Tooling & Machining Association
Judith Alamprese
COSMOS Corporation
February 1997
NATIONAL ALLIANCE OF BUSINESS
ACKNOWLEDGMENTS
This publication was authored by a team consisting of Eunice
N. Askov, Institute for the Study of Adult Literacy at The
Pennsylvania State University; John Hoops, Western
Massachusetts Chapter of the National Tooling & Machining
Association; and Judith Alamprese, COSMOS Corporation. Terri
Bergman, National Alliance of Business, provided guidance in
writing and editing the publication. P. Shawn Jenkins,
Institute for the Study of Adult Literacy, developed the list
of resources in the appendix.
The National Workforce Assistance Collaborative would like to
thank the individuals who provided assistance and reviewed
drafts of the publication:
_ John Barone, Delaware Manufacturing Alliance
_ Dale Brandenburg, Wayne State University
_ Lynne Fry, U. S. Department of Labor
_ Dallas Garrett, Cuyahoga Community College
_ Sandra L. Hodge, Cuyahoga Community College
_ Donna Miller-Parker, Washington State Board for
Community and Technical Colleges
_ Stephen Mitchell, National Alliance of Business
_ Nancy Renner, Catonsville Community College
_ James Van Erden, National Alliance of Business
This publication could not have been completed without their
help.
TABLE OF CONTENTS
Preface 1
This Guide in Context 3
Introduction 5
Designing and Implementing a Training Evaluation 8
1. Establish a Relationship Between the Provider and
Customer 10
2. Gain the Involvement of Key Stakeholders 12
3. Identify the Goals and Objectives for the Training
16
4. Specify the Outcomes 18
5. Examine the Inputs (and Processes) 19
6. Identify the Data Required 21
7. Collect the Data 24
8. Analyze the Data 27
9. Interpret the Data 29
10. Use and Report the Data 32
Conclusion 34
Appendices 35
Training Evaluation Resources 36
National Workforce Assistance Collaborative Advisory
Groups 40
National Workforce Assistance Collaborative Products and
Services 41
PREFACE
Many companies trying to improve their productivity and
competitiveness are turning to training as a means of
achieving this end. Some of these companies, though, are not
willing to accept training's value on faith, and are looking
for proof of training's worth. Assessing the Value of
Workforce Training is designed to meet this need. It provides
an introduction to evaluating a workforce training program,
both to assess its impact and improve its effectiveness.
This Publication Is for. . .
Small and Mid-Sized Companies
_ Individual (or department) responsible for providing
training
And the Training Providers that Serve Them
_ Community colleges
_ Universities
_ State literacy and training programs
_ Labor-management associations
_ Business organizations
_ Trade and professional associations
_ Business development centers
_ Private industry councils
_ Non-profit organizations
_ Private consulting firms
This guide provides instructions for assessing a single
training program, rather than a training department in a
company, or a training provider. It can be used to determine
whether training results match training goals, whether
training dollars are being well spent, and how training
programs might be improved.
Assessing the Value of Workforce Training can help both those
from small and mid-sized businesses responsible for training,
and those from service provider organizations that provide
training (such as community colleges). It can be used for
training evaluations conducted internally by a training
department, or externally by a third-party evaluator.
Assessing the Value of Training is specifically targeted at
small and mid-sized companies and the providers that serve
them. While the material presented here is equally relevant
for larger companies, most of these companies have the
resources to engage in more extensive, rigorous, and costly
evaluations. This guide presents a simplified assessment,
more useable than a full-scale, scientific "evaluation" would
be However, to be consistent with commonly-used terminology,
the guide will usually use the term "evaluation," instead of
"assessing the value," from here on out.
The literature is replete with textbooks, manuals, handbooks,
and guides to evaluation of all types. Rather than
duplicating the many excellent resources that presently
exist, this guide distills the essence of good evaluation
practices and provides guidance for applying these practices
to workplace training. It enables you to "stick your toe in
the water" of program evaluation. Readers who want to gain
greater depth and precision in evaluation, and acquire some
useful tools, can refer to the annotated resources list in
the appendix.
Assessing the Value of Workforce Training is divided into
four sections:
1. This Guide in Context -- Lays out the entire training
process and identifies how and where this guide can help you
in that process.
2. Introduction -- Defines evaluation and explains its
value. To make this guide readable and easy to follow, the
introduction has been structured around a series of
questions.
3. Designing and Implementing a Training Evaluation --
Describes how to design a training evaluation and how data
may be collected and analyzed. This section has been
structured around a series of ten steps.
4. Appendices -- Contains lists of training evaluation
resources, the Collaborative's Board and Councils, and the
Collaborative's products and services.
THIS GUIDE IN CONTEXT
Assessing training's value is just a part of the process of
developing and operating a training program. To ensure that a
training program will meet a company's needs, the company, or
a provider in concert with the company must:
1. Identify the problem facing the company,
2. Determine whether training will solve that problem, and,
If training is called for,
3. Determine the most appropriate methodology for
delivering training,
4. Select the most appropriate training provider or
product, or develop appropriate training,
5. Set up the training program,
6. Deliver the training program,
7. Evaluate the training results, and
8. Use the evaluation results to improve future training
delivery.
This guide only addresses steps 7 and 8:
_ Evaluating training results, and
_ Using evaluation results to improve future training
delivery.
It assumes that:
_ A business problem or objective was identified,
_ It was determined that training would address this
problem or meet this objective, and
_ A training program was selected (or developed) for
delivery.
The National Workforce Assistance Collaborative has developed
a number of other tools to help companies and service
providers carry out the steps involved in developing and
operating a training program, including:
_ Assessing an Organization's Training Needs, to identify
performance assessment issues and help identify assessment
tools,
_ Selecting and Implementing Computer-Based Training,
checklists and text that will enable individuals to determine
the most appropriate 1) training media, 2) type of computer-
based training, and 3) computer-based training product,
provider, or site.
_ Employee Training Product Checklist and Workplace
Literacy Product Checklist, to select the most appropriate
employee training or workplace literacy product,
_ Employee Training Interview Guide and Workplace Literacy
Interview Guide, to select the most appropriate employee
training or workplace literacy provider,
_ Resource Guide, for locating training resources and
providers, and
_ Workplace Literacy Publications, for locating workplace
literacy tools and information.
All Collaborative publications can be ordered through the
National Alliance of Business Distribution Center, listed on
the inside front cover. Collaborative products can also be
downloaded from our Internet gopher server or world wide web
home page. Instructions for accessing these are contained in
the appendix listing Collaborative products and services.
INTRODUCTION
The evaluation of training raises a number of questions. As
you read these questions and their answers, try to apply the
information to your own training situation.
What is evaluation of training? Evaluation is the process of
identifying and measuring the outcomes of a training program
and comparing them to the program's objectives. Evaluation
can also involve identifying the training program's inputs
and processes and linking them to the outcomes, in order to
improve the impact of the training program. In other words,
did we get what we wanted from the training program? How can
we improve it for the next time?1
Evaluating a training program is somewhat similar to
evaluating a restaurant. If a friend asks you how you like
the new restaurant on the corner, your response would
probably be based on the factors you care about, such as the
taste, appearance, and price of the food, the general
ambiance of the restaurant, and the efficiency and courtesy
of the wait staff. These factors are called "outputs," or
"outcomes."
If the restaurant owner is a friend of yours, and asks you
for suggestions on how to improve her new restaurant, you
would want to look at "inputs," or the factors that go into
making a good restaurant, and see how these are linked to the
outcomes. You might, for example, study the freshness and
quality of the food ingredients, the skills of the employees,
and the sanitation of the food preparation, and determine
which of these are the likely causes of any good or bad
results. With this knowledge, you would be prepared to offer
your friend valuable advice.
In evaluating a training program, a similar process is
followed. You would first identify the company's goals for
the training program, and then assess the outcomes of the
training program in relation to these goals. For example, if
a company offers a training program to prepare workers to
create more widgets in a given amount of time, then the
training program should be assessed for its impact on the
productivity of the workers, not on other criteria, such as
safety (as important as this is).
The evaluation should determine the extent to which the
training program met the needs identified by a company needs
assessment. The evaluation should also determine the extent
to which the training was aligned with (or grew out of) the
business or strategic plan. In other words, the training
should address identified needs and goals of the company, and
the evaluation should assess whether the training succeeded.
Most evaluations also have the goal of improving the training
program. To do this, evaluations need to identify the
training inputs (and processes), and link these inputs to the
outcomes (or outputs). Training inputs could include such
things as the background characteristics of the workers being
trained and the skill of the training personnel, as well as
aspects of the training program, such as the training
curriculum and the program delivery systems. Once you have
linked these inputs to the outcomes, you would have a basis
for making improvements to the training program.
Why should small and mid-sized businesses evaluate training?
What can an evaluation tell you? It makes sense to evaluate
any business efforts, including training, that cost a company
time and money. You need to know the impact of the training
program not only on the "bottom line," but also on all
stakeholders in the company, including workers, unions,
supervisors, managers, owners, trainers, and customers. You
also need to know how training might be improved to be more
effective and efficient -- to produce better outcomes.
Training can be one of a company's most important and
valuable assets: the capacity to enhance workers' skills and
achieve company goals. Training can also benefit employees,
increasing their labor market value, improving their incomes,
advancing their careers, and enabling them to grow as people.
All of these may be implicit in a company's decision to seek
training for its employees, and are often explicit in a
provider's efforts at marketing training services.
How do we know that any of these outcomes actually takes
place as the result of training? A good, simple evaluation
will help validate training services and will benefit the
company, the provider, and the workers receiving training. An
effective evaluation will indicate whether or not immediate
goals and objectives were met. It can also pinpoint the
impact of training on the workforce and the company. More
broadly, an evaluation will help suggest ways to improve
future training and indicate ways that the provider and
company can develop a more productive training and skill
development relationship.
What factors influence the evaluation of training? The
evaluation process will vary depending on a number of
factors, some of which are:
_ Specificity of the training program objectives,
_ Number and background characteristics of the workers
involved in the training program,
_ Use of existing measuring devices that assess the impact
of the training,
_ Amount of data related to the training program that the
company already collects, and
_ Willingness of the top management to send the message
that the training evaluation is important and necessary.
A highly specific training program, such as one designed to
teach the use of a particular piece of equipment, may be
shorter in duration than, say, a workplace literacy program.
The results of a highly specific training program will also
be more easily measured than those from a program with the
objective of raising workers' general functioning level. The
impact of a specific training program can be assessed almost
immediately, and outcomes can be directly attributed to the
program.
With a more general training program, such as workplace
literacy, the immediate impact may be hard to determine
because the outcomes may be indirect and only evident over a
longer period of time. The impact of general training
programs can be calculated, though, just not easily. An
outcome such as making workers more trainable through
increased knowledge and skills -- and therefore more able to
participate in later technical training -- can be measured
and evaluated.
The skill level of the target audience for the training will
have an impact on the evaluation instruments you can use.
Workers with low literacy skills may have difficulty with
complex evaluation forms, for example. If you can use
existing instruments and data that are already being
collected, the evaluation process will be less disruptive and
the cost of the evaluation can be greatly reduced.
And, as with any special program, the degree to which both
management and union leaders publicly endorse the evaluation
process will affect the degree to which all levels of the
organization "buy in" to the evaluation process. This buy-in,
in turn, will facilitate the data collection process, as well
as other aspects of the evaluation.
Effective training evaluation should serve as the basis for
improving the quality of a training program and creating
better outcomes. The next section provides information on
designing and implementing a training program evaluation. It
also suggests some good practices that may prevent potential
problems.
DESIGNING AND IMPLEMENTING
A TRAINING EVALUATION
This section identifies some of the key elements and
procedures for designing and implementing a training
evaluation. It is divided into chapters covering a ten-step
process:
1. Establish a relationship between the provider and
customer,
2. Gain the involvement of key stakeholders,
3. Identify the goals and objectives for the training,
4. Specify the outcomes
5. Examine the inputs (and processes),
6. Identify the data required,
7. Collect the data,
8. Analyze the data,
9. Interpret the data, and
10. Use and report the data.
The first five steps explain how to design a training
evaluation, while the last five steps explain how to
implement a data collection plan. The questions that the
company's stakeholders want to have answered by the
evaluation drive the evaluation process, and are the focus of
the first five chapters. These questions should specify the
performance changes expected from the training program, and
should be identified before the training begins.
The questions form the framework of the evaluation plan and
determine the data that will be collected for the evaluation,
the focus of the last five chapters of the guide. The guide
emphasizes using as much existing data, data collection
systems, and assessment tools (such as skill and knowledge
assessments) as possible, in order to simplify the evaluation
process and keep costs down.
This guide is constructed to help providers and employers
plan for evaluation from the outset of a training effort. By
thinking and planning for a training assessment from the
beginning, you will be able to get a richer and more cost
effective evaluation than if you wait until the end of the
program to ask what happened.
Evaluation should be used as a tool for improvement, similar
to continuous improvement programs in companies, where plans,
data, and feedback are used to enhance ongoing programs. This
also argues for being aware of and using assessment tools
from the very beginning of a training effort.
The training evaluation laid out in this guide is based on
two overriding principles:
_ Clarify evaluation goals -- identify the concrete goals,
objectives, and expected outcomes for the training during the
planning effort, and
_ Simplify data collection -- look for existing sources of
data and information that can verify the achievements of
training objectives and outcomes.
How can you plan a good, simple evaluation? Is there such a
thing as a good and simple evaluation? Absolutely. Evaluation
of training delivery is good when it answers several basic
questions:
_ Did stakeholders receive what they had hoped for?
_ What was the impact (outcomes) of the training program?
_ What were the factors (inputs) that made it successful
(or not)?
An evaluation can be simple when the evaluation goals are
clear and when the planning and execution rely as much as
possible on the input of stakeholders and on already existing
data sources.
1. Establish a Relationship between the Provider and Customer
Critical Components
_ Build a relationship
_ Learn about the company
Good training evaluations are built on solid relationships
between providers and training customers, regardless of
whether these are outside providers working with independent
companies, or in-house training departments working with
other departments in their own companies.
Solid relationships are based on trust; and trust is built on
openness, consistent performance, and results. Part of
building a good relationship is the accountability and
responsibility that comes through the development of a good
evaluation strategy. If the relationship between a provider
and company is to be sustained, the company's decision makers
must have confidence in the provider and the provider's
capability to deliver what it promises -- results-oriented
training and quality evaluations that prove those results.
A training provider must learn the company's goals and
objectives, understand its business needs and strategic plan,
and know its workforce and management style if it is to
design an effective evaluation. It needs to learn about the
context in which training will take place: the company's
environment and its reasons for seeking training.
Companies function through a complex set of
interrelationships within their industries and markets. They
must respond to their customers and to competitive pressures.
Companies may seek training in response to either external or
internal pressures. A company may hope to recover a
declining market share or expand into a new market.
Alternatively, it may perceive serious deficits in the skills
of the workforce, skills that were adequate before work
restructuring or the introduction of new technology.
Relationship building should allow the training provider to
begin laying the groundwork for its evaluation strategy. From
the moment it begins working with a company, the provider
needs to build support for the evaluation, pinpoint the
information and outcomes that are important to the company
and the workforce, and start collecting data.
It is essential that evaluators from outside a company form a
close partnership with that company. Third-party evaluators
may not know the company's business goals, or have access to
the data or to the data collection systems that are needed
for the evaluation process. Outside evaluators can play a
valuable role, in concert with companies (and, whenever
possible, training providers), in designing and implementing
evaluations, but they probably cannot acquire all of the
information they need to carry out the evaluations on their
own. Company personnel must help define the program's
objectives, collect data, and interpret the results. This
involvement of company personnel is even more critical when
training outcomes are to be measured over a period of years,
after the consultant is no longer in the picture.
2. Gain the Involvement of Key Stakeholders
Critical Components
_ Identify stakeholders
_ Work with stakeholders
A successful evaluation requires the involvement of key
stakeholders in planning and execution. Stakeholders are the
people who will be directly or indirectly affected by a
training program. These include both participants (the
training recipients) and decision makers (supervisors,
managers) who invest time or money in the training.
The type and position of people who are direct stakeholders
will vary by the size and organization of the company: In
very large companies stakeholders may not include senior
management, while in smaller companies the president or owner
may be involved directly. As much as possible, stakeholder
groups should include individuals from a variety of
positions, and with a variety of roles, to encourage
cross-fertilization of ideas and exchange of information.
Most often stakeholders are personnel within a company,
including workforce representatives in unionized companies
and personnel from the training department, but they can also
include other "outside" stakeholders such as key customers.
Where the training is being carried out at the request of a
company's key customer (e.g., ISO 9000 training required to
qualify as a supplier to a large company), it is logical to
consider including the customer in the evaluation process.
This principle of involvement is a major departure from the
old rules of evaluation in which "objective" outsiders
conducted an evaluation with limited input from the
"subjects." Including key stakeholders, however, can be both
effective and efficient. It draws on the experience and
knowledge of the individuals directly involved in the program
-- those who have an interest in the outcomes. These
individuals want a program that provides quality and value
and meets company objectives. They are, therefore, an
excellent (and available) source of ideas and data.
Tapping into the knowledge of participants (management,
workers, union representatives, trainees, customers, and
trainers) will vastly improve the content, efficiency, and
effectiveness of training as well as the evaluation.
Stakeholder involvement will help clarify the objectives and
anticipated outcomes of the training. In other words,
stakeholders can tell the provider (and each other) what is
important to them and in what ways training resources should
be directed. This information is essential in identifying the
objectives for a training program.
Stakeholder involvement increases the capacity of the company
and workers not only to participate in the evaluation, but
also to make decisions based on the results of the evaluation
process. Through involvement, people develop ownership over
the process and the outcomes of the training. Evaluation is
not something done to them but with them; they have the
opportunity to help shape and improve the efforts.
Stakeholder involvement increases a company's capacity to
take on some training and evaluation tasks itself. It teaches
individuals how to design and carry out their own assessments
of other programs. This, in itself, is a valuable outcome
that can be applied to other areas of the workplace.
A Stakeholder Group
In 1990, the Pneumatic Scale Company of Quincy,
Massachusetts, decided to offer a training program in
machining, electrical assembly, and electronic systems to a
portion of its workforce. The goal was to address quality
improvements and lead-time reduction by increasing the skills
of assembly workers.
A stakeholder group was formed to guide the program and to
assess the inputs and outcomes. The group met regularly
(monthly or quarterly) for nearly two years and took overall
responsibility for the program. The stakeholder committee
included:
_ Representatives of the assembly department,
_ Union shop floor steward,
_ Union president,
_ Shift supervisor,
_ Department foremen,
_ Manufacturing manager,
_ Operations director,
_ Human resources director,
_ Local technical college (provider), and
_ State agency staff (funder).
How do you form the stakeholder advisory group? Because
stakeholders should be involved at every step in the training
and evaluation design, the provider needs to take time at the
outset to identify the appropriate stakeholders, form a
group, and build in meeting time for structured discussions
on program design, curriculum, delivery, and organization.
Shared governance of the advisory group is critical. In other
words, one set of stakeholders, such as management, should
not dominate the meetings.
Forming the group and getting it off to a good start is no
trivial matter. The stakeholder advisory group should be
formed at the invitation of management; people need to know
that management approves of the activity and sees it as a
valuable use of time. The provider should draw up a list of
key stakeholder categories, and then have appropriate company
personnel (including union representatives) review the list
and suggest names for invitation to the stakeholder group.
Invitations to people outside of the company should also be
issued by company management.
How do you use the stakeholder advisory group to shape the
evaluation? Stakeholders need to be involved at all points in
the evaluation process, both to gain their "buy-in," and to
make the results as useful as possible. Some of the
stakeholders, such as front-line supervisors, might actually
be needed to gather data for the evaluation.
It should be made clear from the outset of calling the
stakeholders together that their purpose is to guide the
training evaluation process. Stakeholder groups are not just
general discussion groups, they have a specific
responsibility: to serve as a resource for the training
provider and the company in the evaluation process. It is
leadership's responsibility to keep the group focused during
the meetings and to help the group understand the importance
of its task.
Stakeholder meetings may be expected and routine in companies
that already have work cell or problem solving teams. In
other companies, providers must ask for the meetings and
build time for them into the basic training design. While
there is a cost associated with this activity, you should
help management understand that the meetings are an essential
component of training delivery.
Elements of Successful, Productive Meetings
_ Meetings should be relatively short and well organized,
and should always have an agenda and structure.
_ Meetings should have a "scribe" who takes notes and
helps the group clarify decisions by using a flip chart,
letting the group decide on the wording of key decisions
(such as objectives).
_ Meetings should also have a chair who promotes
participation, and helps the group move along and follow the
agenda.
In companies that do not generally work in teams or hold
meetings, an initial stakeholder advisory group meeting may
be used to offer training and practice in effective meeting
techniques and group decision making.
When stakeholder advisory groups work well, they very nearly
guide themselves and provide their own internal leadership
and direction. At first, however, groups will need a clear
mandate and some guidelines.
The basic mandate of a stakeholder group focused on
evaluation is to help the provider and the company determine
the effectiveness and impact of the training program. The
group needs to come to a common understanding on some of the
following questions:
_ Why should we do training or participate in a training
program?
_ How does training fit into the company's business or
strategic plan?
_ What do we (a question for the company, for the union,
and for individuals) want to gain or accomplish from the
training program?
_ What are the best methods and format of delivery?
In terms of an evaluation, the two most important questions
are:
_ What can we expect from training?
_ How do we measure the results?
The discussion and the answers to these questions, which need
to be asked and re-checked during the training program, will
lead the provider and the company to a clear understanding of
the impact and value of training. They will also help the
stakeholder advisory group participate in the evaluation
process in an objective, rather than subjective (and less
useful), way.
Keys to Success
_ Involve all people affected by the training program:
management, workers, and stakeholders outside of the company.
_ Provide training neeeded to help the group be effective
and efficient.
_ Keep the group focused on the task at hand: guiding and
facilitating the evaluation process.
3. Identify the Goals and Objectives for the Training
Critical Components
_ Identify goals and objectives
_ Build specificity
A critical next step in the evaluation process is to identify
the goals and objectives of the training program. Why,
specifically, is this particular training being done? Why
now? What do the management or workers hope to accomplish?
What is driving the need for training?
Answers to these questions often begin with general
statements; they need to be more specific for the purposes of
evaluation. Statements like
_ "We want a skilled workforce to help our company be more
competitive," or
_ "It is important for all of our workers to read"
are too general. Goals and objectives will need to be tied to
measurable and observable outcomes associated with the
training, as well as aligned with the company's business or
strategic plan.
A concrete statement that describes the training, the reasons
for the training, and the specific results expected from the
training is most helpful. For example, "company X wants to
train machine operators in preventive maintenance techniques
in order to reduce machine failure and machine downtime." Or,
"company Y wants to provide problem solving skills to
assembly operators in order to decrease the incidence of
errors and rework." Or, "company Z purchased an upgraded
Computer Aided Design package and needs to train its
engineers to use the system."
Often, a smaller firm's external customers drive changes
requiring new technology or new procedures; the rush to ISO
9000 in manufacturing is such an externally driven change.
When this happens, firms will seek training for their workers
in order to satisfy and retain customers. In this case, as in
others, meaningful training must be tied to a specific
business objective.
Achieving specificity on goals and objectives can be
relatively complicated, especially for broad, comprehensive
skill training programs. Using stakeholders to help specify
goals and objectives can make the process much simpler. It is
very important, however, to help the stakeholders develop
clear goals and objective statements. Draft statements should
be written down, circulated among the stakeholders, and
refined to a consensus position. The goal and objective
statements can then serve as guideposts for the training and
evaluation program.
For example, a statement of goals and objectives for training
could be written as follows:
Our department will provide machine maintenance training to
machine operators in order to reduce the frequency of machine
breakdown, lower the costs of machine maintenance, increase
machine utilization, and thereby increase production. Through
this training program, machine operators will learn to
identify signs of potential machine failure, and learn
procedures for routine machine maintenance and simple repairs
that can be performed without calling machine maintenance
personnel. Success of this program in our department will
contribute to lower operating costs for the company.
This statement identifies specific goals and objectives for
the training. It states
_ What will take place -- machine maintenance training,
_ Results expected within the department -- reduced
breakdowns, lower maintenance costs, and increased machine
utilization and production,
_ Results expected for the workers receiving training --
ability to identify potential failure and perform routine
maintenance and simple repairs, and
_ The expected impact on the company -- lower costs of
operation.
Keys to Success
_ Involve the stakeholders in identifying the goals and
objectives.
_ Be as specific as possible. Goals and objectives will
need to be tied to measurable and observable outcomes.
4. Specify the Outcomes
Critical Components
_ Determine outcomes
_ Focus outcomes
A clear statement of the goals and objectives guiding the
training will help target the outcomes that the company and
the provider expect to observe as the result of training.
Outcomes become the evidence that the training did or did not
have the desired effect. They are the basic accomplishments
of the program. As we stated earlier, outcomes will occur at
an individual level (a machinist increases speed and accuracy
in CNC programming) and at an organizational level (the CNC
cell reduces lead time). Both of these types of outcomes are
important to identify and measure.
Obviously, then, it is to everyone's benefit to spend some
time at the beginning of the training relationship thinking
about specific outcomes and the ways that the outcomes can be
easily and simply observed. One meeting of stakeholders
brainstorming about potential training outcomes can generate
a very long and comprehensive list. The question to ask is:
What will happen (to a job, to a process, to a person, to a
product, to the company) if this training is successful?
Increasing the capability of machine operators to perform
preventative maintenance on machines, for example, may
directly affect operators' skills and knowledge about the
equipment, product flow, product quality, and operation
costs. Which of these are the most important outcomes? Which
outcomes reflect the goals of the individual (more knowledge,
greater promotability, increased scope of action and
flexibility) or the company (higher product quality, less
machine downtime, lower maintenance costs)?
Keys to Success
_ Consider both individual and organizational outcomes.
_ Use stakeholders to brainstorm potential outcomes.
_ Focus the evaluation on the most critical outcomes.
5. Examine the Inputs (and Processes)
Critical Components
_ Determine individual inputs
_ Determine organizational inputs
In order to determine whether efforts to respond to customer
needs are effective, you have to assess what goes into the
training as well as what comes out. What goes into the
training are known as the inputs to training.
Inputs to training, like outputs, may be both individual and
organizational. Individual inputs include the background
characteristics of the trainees, while organizational inputs
pertain to the company and its training program. These
organizational inputs include the company environment in
which the training program is offered, as well as the
planning, curriculum, and delivery methods of the training.
In traditional training environments and among high-volume
commercial training providers, training planning and delivery
are determined by the provider. Some educational
institutions' curriculum and delivery may not be sufficiently
flexible or responsive to customer needs. However an
increasing number of providers are incorporating flexibility
of format, content, and delivery into training design
Inputs can have a significant impact on training results. For
example, training may not produce expected outcomes simply
because it was not targeted to the workers appropriately.
Some quality improvement programs assume a high level of
literacy and reading comprehension on the part of the
workforce which might not be present. The literacy level of
the workforce can be considered an "input" which should be
taken into account in the evaluation of the training.
Organizational inputs also affect training results. For
example, a training program could have been offered at an
inconvenient location or time for its target group. If
participants feel that the training program is not valued
because it is offered in the basement, or if participants
can't attend due to conflicting schedules, then the training
will not produce the desired outcomes no matter how good it
is.
Ideally, an effective stakeholder advisory group would
carefully identify key conditions (inputs) for training
success and solve issues such as scheduling before the
training even began. In the real world, however, things are
more complex. A stakeholder advisory group can help the
evaluation be sensitive to changing conditions -- in the
workforce, the company, and the provider -- that may have
affected training results.
Keys to Success
_ Develop a list of all the individual and organizational
inputs that could positively or negatively affect training
results.
_ Use the stakeholders to keep a pulse on changes in the
workforce, company, or provider that might affect the
training program.
6. Identify the Data Required
Critical Components
_ Identify outcome measures
_ Identify input measures
Once the expected outcomes have been clearly specified and
the inputs considered, data must be found to measure these.
Data are most often thought of as quantitative information,
such as cost or units of numerical measurement (time, for
example). But other types of data are also important.
Qualitative data, which might result from interviews with key
participants, or anecdotal evidence, can be equally important
in assessments.
Some of the questions about performance changes (outcomes)
will be answered by individual data; some by organizational
information. Likewise, data related to inputs are both
individual and organizational. The following table summarizes
the evaluation process with some sample entries.
EVALUATION PROCESS
Individual Outcomes
What to Measure
How to Measure
Learning gains
Mastery tests, standardized skill and knowledge assessments
(e.g., occupational skill standards assessments), alternative
assessments (e.g., portfolio assessments)
Attitudes
Surveys, interviews
Skill transfer to the job, changes in job performance
Job output, supervisory ratings, interviews, tracking
promotions, observation
Organizational Outcomes
What to Measure
How to Measure
Productivity, quality, reduced waste, safety
Existing company data collection systems
Individual Inputs
What to Measure
How to Measure
Trainees' background characteristics
Employee records, questionnaires, skill assessments
Organizational Inputs
What to Measure
How to Measure
Company environment (e.g., hiring/downsizing, support for
training, work processes)
Company data, questionnaires, interviews, observation
Amount of training
Training records, questionnaires
Training content
Curriculum review
Training delivery
Questionnaires, participant logs
Training costs
Available data
You can use this table to design your own training
evaluation. Use the outcomes and inputs identified in earlier
chapters to define "what to measure." Then think about
potential data sources for measuring the outcomes and inputs.
Start with a blank table and fill it in with entries relevant
to your training program.
If the outcomes you are interested in are whether people are
more satisfied with their jobs or feel that their skills or
job environment has improved, then people's perceptions are a
very good source of data. For the most part, however, a
company's management will look first at such quantitative
outcome measures as time saved, productivity increased, or
quality improved.
What types of measures can you use to assess change? Since
the questions for the evaluation plan are developed by the
stakeholders, the stakeholders can also help identify the
information that will provide answers to the questions. It is
best to determine up front, with the stakeholders, the
minimum information needed to answer the questions about
outcomes and inputs. This information would be entered in the
table column entitled What to Measure.
Individual outcomes from training usually include learning
gains, improved attitudes, enhanced performance, and
promotions. Transfer of the knowledge and skills acquired in
the training program is really "the proof of the pudding."
Evidence of positive organizational outcomes from training
might include:
_ Sales records,
_ Production rates,
_ Waste reduction rates,
_ Customer satisfaction,
_ Safety statistics, and
_ Retention and absentee rates.
Reviewing data and evidence with the stakeholders will also
help the evaluator identify the types of data that are
important to different stakeholders. For example, while very
few companies keep extensive skill assessment or performance
data on individuals, some stakeholders may feel that one of
the most valuable measures of a training program's
effectiveness is changes in individuals' skills or
performance. Therefore, collection of new data may sometimes
be warranted and considered worth the time and expense.
Strategies Organizations Employ for Evaluating Training
Type of Evaluation Percentage Using
Measuring trainees' reactions 86%
Testing trainees 71%
Evaluating trainees' behavior 65%
Measuring business results 49%
-- "1996 Industry Report," Training, October 1996
The box showing the percent of organizations using various
strategies for evaluating training indicates that most
organizations focus on documenting individual outcomes,
especially measuring how people "feel" about training. While
this strategy is the easiest to implement, it provides the
least amount of information about the actual impact of the
training program on both individuals and the company.
Keys to Success
_ Use earlier work defining outcomes and inputs to
determine "what to measure."
_ Let stakeholders determine the key items that have to be
measured.
7. Collect the Data
Critical Components
_ Access available data
_ Use existing assessments
_ Design new data collection instruments
In developing the evaluation plan, you should consider the
type of data collection that is feasible for a company's
organizational structure and resources. Data required for an
evaluation can be broken down into two categories: data that
already exist and data that do not exist. The latter, all
joking aside, may be expensive and require a great deal of
effort to collect.
As much as possible, it is important to first look at
existing data to see if they will answer the questions posed
by the training evaluation design. If the existing data will
answer the questions posed, then the evaluation process can
be relatively inexpensive.
The appropriate data, however, may not exist or be available
to answer the evaluation questions. If this is the case, the
evaluation will require extensive (expensive) data collection
procedures. The cost of this endeavor must be weighed against
the value of the evaluation. It might be appropriate to offer
the training program without extensive evaluation to avoid
the expense of collecting new data.
It most cases, some relevant data will be easily accessible.
Many smaller companies, although nearly overwhelmed by
requirements to provide information to their customers and to
local, state, and federal regulatory agencies, already
collect a lot of data on such variables as quality,
production costs, value added per employee, throughput,
scrap, administrative costs, lead time, inventory, material
turns, product flow, and cost of sales. These data are often
useful in conducting an evaluation of a training program.
If the company is already using the data for cost analysis,
planning, or forecasting, then these data are relevant and
important to effective management. Tying the evaluation to
this existing information will not only make the evaluation
easier to conduct, but will also make it more relevant and
important to the company. The evaluation may help the company
look at its existing data and information in new ways that
provide new insights into company operations.
How can you collect information to assess change? It is
important that the data collected actually answer the
questions being asked. At times, data may be collected
because they are accessible, even though they don't directly
pertain to the questions being asked. As a result, too much
data can be collected and the process can become more
burdensome than it needs to be. You should work with the
stakeholders to decide how to collect the necessary data and
enter the information in your table column entitled How to
Measure.
Trainees' learning gains should be assessed by specific tests
that indicate whether or not a person has mastered the
knowledge and skills taught in the training program. Should
the program evaluation then use skill testing? Stakeholder
advisory groups can best answer this question and weigh the
relative costs and benefits.
The trainer is usually the best person to devise specific
tests, since the tests should measure mastery of the training
program content. Sometimes a standardized test may be used to
measure learning gains. Before using one, though, you should
first be sure that the test measures information needed to
answer the questions posed in the evaluation design. If the
test has been properly developed and field tested, you can be
confident that the results will be accurate.
Transfer of knowledge and skills learned in the training
program to the job may be assessed in a variety of ways:
_ Supervisor rating scales measuring job performance. (The
rating scales should focus on application of the specific
knowledge and skills taught by the training program, rather
than on global job performance.)
_ Work samples collected over time to identify inclusion
of newly learned knowledge and skills.
_ Observation scales measuring the specific aspects of job
performance that relate to the training program.
_ Interviews determining employees' and supervisors'
perceptions of the effects and quality of training. (You
should look for themes that emerge from open-ended responses
so that you can determine the perceived impact and quality of
the training program.)
_ Written surveys soliciting reactions to the training
program in structured questions and open-ended responses.
Various stakeholders, such as front-line supervisors, may be
able to suggest ways to assess the transfer of skills and
knowledge learned in training to the job. They may also be
able to collect these data.
Existing data may be the best source for assessing changes in
a company due to the training program. It might be difficult,
however, to specifically tie some of these to the training
program since they are influenced by multiple factors, such
as the time of the year in businesses affected by seasonal
trends. Individuals in the stakeholder advisory group, who
represent many different aspects of a company, are probably
in the best position to identify which organizational
indicators of change in the company might be accurately
credited to the training program.
Similarly, existing data may also be useful in considering
the inputs to the training program. Again, the stakeholder
group is the best source of information about the inputs that
may have influenced the outcomes of the training.
What are other considerations in collecting data? Sometimes
you may have to design a new data collection instrument. When
doing this, you must be careful to establish that the
instrument really does measure what you intend. Asking the
stakeholder advisory group to review your instrument can help
ensure its relevance.
You also must be sure that your instrument will give you
consistent results. You should first try it out with some
workers who are similar, but not the same as, those in the
training program, to see if they answer the questions as you
expect. If certain questions on your test are frequently
answered incorrectly, check them for ambiguous wording. You
may also want to review the answers with employees who took
your pilot test to see why they answered as they did so that
you can eliminate poorly worded items.2
Other considerations in collecting data are:
_ Following test administration procedures if you are
using standardized tests,
_ Setting consistent data collection procedures for newly
designed instruments, and
_ Training data collectors, especially if more than one
person is administering the instruments, to assure accuracy
and consistency of the data collected.
Keys to Success
_ Use existing data as much as possible, but don't use
data that isn't pertinent.
_ Look for good assessment tools that measure the outcomes
and inputs relevant to your evaluation.
_ Make sure that any data collection instruments you
develop are both valid and reliable.
8. Analyze the Data
Critical Components
_ Use "control group" analyses
_ Use "pre-post" analyses
The major purpose of the evaluation process is to determine
whether or not change occurs as a result of the training
program. Measuring that change requires a comparison of
performance after training to:
_ A similar group that did not receive the training
("control group"), or
_ Baseline data, such as pre-tests, that are gathered
before training.
In an ideal evaluation design, you would collect data from
the employees participating in the training and from a group
of comparable employees who are not involved in the training
program (a control group), and compare the status of the two
groups after training. There could be changes in performance
in both groups. However, any difference in the changes
between the two groups could be attributed to the training.
This type of comparison ensures that outcomes observed
following training are due to the training intervention and
not to something else.
If this "control group" design is not feasible (due to cost,
difficulty of withholding treatment from some individuals,
etc.), a pre-post design should be used to view changes over
the course of the training. Baseline data would be needed to
judge whether the training program has an impact on
individuals. To measure learning gains, for example,
individuals could be administered pre- and post-tests related
to the training. Baseline data can also be useful in
measuring organizational outcomes. A company would need to
know its safety record before training takes place in order
to determine any change after a safety training program.
In using this design, you must determine the amount of time
between the pre and post data collection periods. Common
sense says that there must be enough time between a pre- and
post-tests for measurable change to occur. In measuring
changes in job performance, it is important to consider the
time it would take for employees to use the skills they have
learned on the job before scheduling data collection.
You can display quantitative data to show pre- and post-test
scores for each participant (presented numerically to
preserve anonymity). You can also aggregate the data using a:
_ Mean -- average score,
_ Median -- middle score, or
_ Mode -- most frequently occurring score.
A mean score can be influenced by extreme scores; however, it
can be used in statistical calculations. If you have a
distribution without extreme scores, the mean may be the most
useful measure. (You must remember, however, that only in
Lake Wobegone are "all the children above average"!) If you
have extremely high or low scores, a median or mode may
provide more relevant information.
For qualitative measures, such as interviews or open-ended
questions on surveys, you will want to analyze the ways in
which respondents' answers change from before to after the
training program. You can look for themes that emerge and
document the number of occurrences.
Keys to Success
_ Use a "control group" design whenever possible.
_ Make sure the time between pre and post measurements is
appropriate for the training and evaluation design.
_ Don't shy away from qualitative measures with either
"control group" or "pre-post" analyses.
9. Interpret the Data
Critical Components
_ Determine causation for individuals
_ Determine causation for organizations
The last step identified whether or not change occurred; this
step determines the causes of the change. The last step
focused on outcomes; this step identifies the inputs that
brought about the change.
With the results of the data analysis (outcomes) in hand, you
need to check the training inputs. If the results were
positive, you must determine whether the training program
brought about the changes, or whether the changes were due to
chance or some other confounding factor (for example, pay
raises to the trainees).
If the training program did not produce positive results, or
even produced negative results, you need to determine why.
Was it the fault of the training program, or of other outside
events? A review of the inputs could suggest the source of
the problems. It may be that the trainees lacked the
appropriate prerequisite skills to benefit from the training
program. Or perhaps the training program was not long enough
or not well attended due to scheduling conflicts. Answers to
these questions can be used to improve the training program
so that it will have greater impact when offered again.
How can you determine whether training caused a change in
individuals? The design of the data collection will influence
the types of inferences you can make about training. If you
have used a matched control group that did not receive the
training program, it is reasonable to assume that any
differences in participant outcomes between the two groups
are attributable to the training.
If you used a pre-post design, you will be able to see the
changes that occurred between the beginning and the end of
the training, but you can't be sure these were caused by the
training. There may be alternative explanations as to why
these changes occurred. You need to consider the confounding
factors, such as other company initiatives or in-house
training, that may account for these differences.
One way to determine whether changes were the result of a
training program is to compare the data on employees'
participation in the training with the outcome results.
Individuals who attended rarely should show little or no
growth as a result of the training, while those who attended
regularly should show improvement. If your results follow
this pattern, they were probably caused by the training. If
they don't, changes you observe may be the result of other
factors.
How employees feel about training and the way training is
treated within a company can significantly affect training's
impact on individuals. Trainees who see personal as well as
corporate benefits to the training program usually attend
voluntarily. Those who are rewarded for participation by
increased wages have higher morale, which may pay off to the
company in greater productivity.
Sometimes, however, training is not voluntary. Workers may be
held accountable for production schedules that do not
recognize their time in training. Supervisors and co-workers
may resent trainees' time off the job and may make their jobs
difficult. Participation in training may not be rewarded by
recognition or increased pay. In these cases, morale is
likely to drop, perhaps negatively influencing participants'
productivity.
Stakeholders can play a valuable role in interpreting
training outcome data. They can examine the preliminary
results from the data analysis, and help identify any
confounding factors that might have influenced participants'
performance on the outcome measures. They also may be able to
identify aspects of the training program that could be
improved.
Interpreting Training Results at a Seattle Hotel
A hotel in Seattle introduced a workplace literacy training
program in order to improve the English language skills of
its housekeeping and food service staff. The hotel expected
that the workers' improved language skills would enable them
to interact better with hotel guests, and that this would
lead to an increase in guest satisfaction. The hotel
calculated a guest satisfaction index for the year prior to
the training program, and then calculated the index again
after the training program. When the hotel compared the two
figures, it noted a substantial gain.
The hotel did not, however, stop the analysis there. It also
compared the change in the guest satisfaction index with that
of a comparable hotel, in the same chain, in another city,
for the same time period. At both hotels the guest
satisfaction index rose, but it rose at a higher rate at the
Seattle hotel that had implemented the workplace literacy
training program. Because the Seattle hotel with training was
able to compare its progress to a comparable hotel without
training, it could feel confident that training caused its
satisfaction index to rise higher than it would have
otherwise.
How can you determine whether training caused a change in
organizations? Determining whether change in an organization
is due to a training program or some other event is much the
same as determining whether training brought about a change
in individuals. If changes at an organization with training
can be compared to changes at a comparable organization
without training, then any differences between the two
organizations can be ascribed to the training program. Such
an analysis, however, is seldom possible.
Organizational impact is very difficult to assess in most
companies because it is even harder to show causation here
than it is with individuals. Collecting relevant baseline
data before training will enable you to make comparisons
after training, but this will not prove causation.
In addition, positive training outcomes can be masked by the
negative effects of other events. Sometimes outcomes from
training can be negated by such factors as poor raw
materials. Sometimes working conditions, such as extreme heat
in plants without air conditioned during hot weather, can
negate the immediate effects of training. Again, the
stakeholders may be able to identify any confounding factors.
In conducting an evaluation, you should also look for
unanticipated consequences beyond the expected outcomes you
identified ahead of time. Capturing unanticipated
consequences, whether positive or negative, is a valuable
component of an evaluation. These consequences can help to
establish the value of training as well as identify changes
or improvements that can be made in planning and delivering
the training program.
The purpose of a training program, for example, may have been
to reduce lead time in a company's machining department. An
unanticipated consequence, resulting from interactions in the
training sessions, may be improved communication processes
between the order entry department and the manufacturing
floor. Without an evaluation plan, this unanticipated
positive impact might not be attributed to the training
program.
Keys to Success
_ Question results. Try to determine whether training was
the cause of individual and organizational outcomes.
_ Let stakeholders help you identify any confounding
factors that might have influenced training results.
_ Be open to unanticipated outcomes. Training might
produce side benefits that neither you not the company had
expected.
10. USE AND REPORT THE DATA
Critical Components
_ Prepare a report
_ Improve the program
Once you've interpreted your data, you need to decide what to
do with the information. Whether the evaluation is conducted
internally by the training department or externally by a
third-party evaluator, a report needs to be prepared. It
should include not only the numerical findings but also a
summary and commentary on the results. Recommendations for
changes in the training content, delivery, and workforce
participation are appropriate if they are based on the
evaluation results.
In business, it would be considered poor practice not to
respond to the legitimate concerns of customers as soon as
they were known. In program operation, ongoing evaluation
should be used in the same fashion: The evaluation process
should be used to continuously improve both the process and
products of a training program.
Continuous improvement is a tool employed by many companies
that uses evaluative data to modify and improve ongoing
practices. Similarly, information and feedback from
participants and stakeholders should be used to gauge and
improve the delivery and organization of training during the
course of the program. This increases the likelihood of
success, as well as the likelihood that there will be a next
program.
Baltimore Center for Port-Related Industries
The initial training class offered to ILA 953 was "An
Introduction to Maritime Industry --Maryland, a Quality
Port." This four-week course introduced the concept of the
port as a single industry, and highlighted the
interdependencies among key players in the port community.
The message in this initial course was so powerful that
instructors decided to reinforce the quality concept in the
planned technical classes.
The staff elected to eliminate "Introduction to Maritime
Industry" as a separate class and to adopt team teaching in
order to integrate the quality elements of the maritime
course into the planned technical classes, and make quality
an integral part of employee work practices. This flexibility
to rewrite curriculum exemplifies the concept of continuous
improvement
Continuous improvement applied to program planning and
delivery confounds traditional program evaluation techniques,
which assume that planning, delivery, outcomes, and
evaluation come in a fixed sequence. In real life, companies
must do their best to improve things as they go along. Their
success in these efforts can be measured just as well, but it
requires a more flexible measurement strategy focused on the
goal of providing effective training.
In fact, through group processes and good meeting practices,
it is possible to be both flexible about change and rigorous
in measuring the impact of change. Feedback from stakeholders
comprises a source of data on the planning and delivery
processes. These data can be charted and the changes in
planning and delivery noted.
Providers can look at data collected on outcomes, and
evaluate these against the changed inputs, in order to
determine training effectiveness. Documentation in meeting
minutes of group decisions and changes in training planning
and delivery will assist the evaluation process.
Keys to Success
_ Don't let the desire for a "pure" evaluation prevent you
from improving the training program whenever you can.
CONCLUSION
Evaluation is a necessary and valuable part of any training
program. The ten steps described in this guide can help you
develop a cost-effective evaluation that involves a
stakeholder group in all aspects of the process.
Some companies are concerned about cost-benefit analysis,
also called return on investment (ROI). The many formulas
available for determining ROI are beyond the scope and intent
of this guide. Those interested in ROI, however, could
consult the series of articles by Phillips (Training and
Development, 1996) that are summarized in the Training
Evaluation Resources Appendix.
Before undertaking an ROI analysis, however, you must
consider whether the time and expense involved is worthwhile.
You may find that only selected training programs, perhaps
those that are repeated frequently or involve most workers,
should be evaluated for ROI. In other cases, the effort and
resources that would be expended in determining ROI might be
better spent supporting your training program.
Evaluation, as described in this guide, can provide important
information about the impact of a training program and
strategies for improving it. Evaluation can provide evidence
of training's impact on individual trainees and on the
organization, information valuable for both the company and
the training provider.
For an outside training organization, evaluation can be part
of its marketing strategy to other companies. Positive
evaluations of previous training programs are proof that the
trainers are knowledgeable and effective. Evaluation provides
the evidence.
APPENDICES
Training Evaluation Resources 36
National Workforce Assistance Collaborative Advisory Groups
40
National Workforce Assistance Collaborative Products and
Services 41
TRAINING EVALUATION RESOURCES
Basarab, D. J., Sr., and Root, D. K. (1992). The training
evaluation process: a practical approach to evaluating
corporate training programs. Norwell, MA: Kluwer Academic.
Discusses formative and summative evaluation. Categorizes
Kirkpatrick's four evaluation levels, with levels one and two
being formative, and levels three and four being summative.
Explains the five phases of the training evaluation process.
Could be used as a working manual for training evaluation.
Charts, tables, instruments, sample evaluation plan, and
sample final report could serve as models for designing
evaluation.
Brinkerhoff, R. O., Brethower, D. M., Hluchyj, T., and
Nowakowski, J. R. (1983). Program evaluation: a
practitioner's guide for trainers and educators (Design
manual and Sourcebook/Casebook). Hingham, MA: Kluwer Boston,
Inc.
The sourcebook divides evaluation into seven areas: 1)
focusing and 2) designing the evaluation; and 3) collecting,
4) analyzing and interpreting, 5) reporting, 6) managing, and
7) evaluating information. Provides a thorough coverage of
this approach to evaluation. The casebook and design manual
may be used with the sourcebook. The design manual offers a
step-by-step procedure for working through an evaluation; it
includes model worksheets and checklists.
Broad, M. L. (1992). Transfer of training: Action-packed
strategies to ensure high payoff from training investments.
Helps managers, supervisors, and employees in U.S.
organizations achieve full job performance by acquiring and
applying effective strategies to transfer the knowledge and
skills learned in training to the workplace. Part 1 covers
human resources development and the transfer problem,
barriers to and management of transfer of training, and key
roles and times to support transfer. Part 2 presents transfer
strategies for use before, during, and after training, and an
action-planning relapse prevention strategy. Part 3 discusses
using and supporting transfer in organizations. The
appendices include strategies for managing the transfer of
training, behavioral processes underlying successful transfer
of training, the steps entailed in conducting a transfer
audit, a glossary, and a bibliography.
Center for the Study of Evaluation. (1987). Program
evaluation kit (2nd ed.) (Vols. 1-9). Newbury Park, CA: Sage.
This evaluation kit has nine volumes (approximately 150 pages
each):
Vol. 1. The Evaluator's Handbook, by J. L. Herman, L. L.
Morris, and C. T. Fitz-Gibbon.
Vol. 2. How to Focus an Evaluation, by B. M. Stecher and W.
A. Davis.
Vol. 3. How to Design a Program Evaluation, by C. T. Fitz-
Gibbon and L. L. Morris.
Vol. 4. How to Use Qualitative Methods in Evaluation, by M.
Q. Patton.
Vol. 5. How to Assess Program Implementation, by J. A.
King, L. L. Morris, and C. T. Fitz-Gibbon.
Vol. 6. How to Measure Attitudes, by M. E. Henerson, L. L.
Morris, and C. T. Fitz-Gibbon.
Vol. 7. How to Measure Performance and Use Tests, by L. L.
Morris, C. T. Fitz-Gibbon, and E. Lindheim.
Vol. 8. How to Analyze Data, by C. T. Fitz-Gibbon and L. L.
Morris.
Vol. 9. How to Communicate Evaluation Findings, by L. L.
Morris, C. T. Fitz-Gibbon, and M. E. Freeman.
Volume 1 has an overview and is a directory for the other
manuals. It contains step-by-step procedural guidelines for
conducting both formative and summative program evaluations.
Each of the other volumes provides elaboration, explanation,
and background information for each aspect of the evaluation
process. Except for The Evaluator's Handbook, each book can
be used individually when focusing on a particular section of
the evaluation process. The Evaluator's Handbook covers
formative and summative evaluations, including setting the
boundaries of the evaluation; selecting appropriate
evaluation methods; collecting and analyzing information; and
reporting findings; and provides an example of an evaluation
report.
Human Resources Development Group. (1994). Establishing the
value of training: Resource guide (HRDG Publication No. HRDG-
023). Washington, DC: United States Office of Personnel
Management, Office of Research and Information.
Provides a clear and concise tool to use when establishing
the value of training. Based on Kirkpatrick's four level
evaluation model, a four stage evaluation process is utilized
including: focus, plan, implement, and evaluate. Provides
step-by-step procedures for doing training evaluation, but is
primarily focused on calculating training's value. Tables and
job aids (sample worksheets, checklists, etc.) are included.
Covers the what, why, when, and how of establishing the value
of a training program.
Joint Committee on Standards for Educational Evaluation.
(1994). The program evaluations standards: How to assess
evaluations of educational programs (2nd ed.). Thousand Oaks,
CA: Sage.
Discusses thirty standards which are divided into four
attributes of an evaluation: utility, feasibility, propriety,
and accuracy. Describes each standard, gives guidelines to
help evaluators meet the requirements of the standard,
relates what difficulties are associated with each standard,
and includes a case study or two to illustrate the
application of the standard. In order to assist in applying
the standards, a functional table of contents reorganizes the
list of standards into ten headings.
Kirkpatrick, D. L. (1994). Evaluating training programs: The
four levels. San Francisco, CA: Berrett-Koehler.
Gives an in-depth look at Kirkpatrick's four levels of
evaluation. Provides guidelines for each of the four levels
of evaluation and elaborates each guideline. Discusses
reasons for evaluation and how to implement the four levels.
Contains case studies of implementing evaluation of training.
Manly, D. (1994). Workplace education evaluation checklist: A
tool for assessing and improving performance. Madison,
Wisconsin: University of Wisconsin, Center on Education and
Work.
This book is an evaluation instrument with no real text. Four
checklists can be used to evaluate workplace education
programs: program design and administration indicators,
participant achievement indicators, program outcome
indicators, and evaluation tools. Checklists could be useful
for the evaluation of other types of training and education
programs if modified.
Medsker, K. L., and Roberts, D. G. (Eds.). (1992). American
Society for Training and Development trainer's toolkit:
Evaluating the results of training. Alexandria, VA: ASTD.
One of two toolkits developed by ASTD. Focuses on levels
three and four of Kirkpatrick's model. Explains how in
combining a model by Nadler with Kirkpatrick's model you are
able to see where Kirkpatrick's four levels come into use.
Renames Nadler's five stages as: needs analysis, preparation,
human resources intervention, on-the-job, and outcomes.
Recommends Phillips' 18 step model also as a standard.
Discusses some basic models and includes examples,
worksheets, tables, checklists, and case studies.
Phillips, J. J. (1991). Handbook of training evaluation and
measurement methods (2nd ed.). London: Kogan Page.
Mentions six models besides Kirkpatrick's model, including
the Bell System approach; the CIRO (context evaluation, input
evaluation, reaction evaluation, and outcome evaluation)
approach; Saratoga Institute approach; the IBM approach;
Xerox approach; and CIPP (context, input, process, and
product) approach. Explains an eighteen-step compete results-
oriented HRD model. Evaluation instrument design, including
tests, questionnaires, surveys, interviews, etc., is
discussed. Chapters cover evaluation design, costs and
returns, data collection and analysis, and using computers in
the evaluation process.
Phillips, J. J. (1996). How much is the training worth?
Training and Development, 50, (4), 20-24.
This is the third article in a series about return on
investment. It shows how to convert program results to
monetary benefits that bosses can understand. It includes a
five step conversion process.
Phillips, J. J. (1996). ROI: The search for best practices.
Training and Development, 50, (2), 42-47.
This is the first in a series of three articles about
measuring the return on investment in training. Provides case
studies to look at how one measures return on investment
(ROI) in training. Includes a framework for developing ROI
and emphasizes the need for a variety of evaluation plans.
Phillips, J. J. (1996). Was it the training? Training and
Development, 50, (3), 28-32.
This is the second article in a series about return on
investment. It describes ten ways to isolate the effects of
training so that training is credited over other variables as
the reason for performance improvements.
Smith, M. E. (1991). Summative evaluation. Performance
Improvement Quarterly, 4, (2), 35-58.
Discusses summative evaluation, focusing on eight steps
involved in the planning process. Topics discussed include
client commitment to evaluation; evaluating the training
organization; evaluating management training, including
appropriate evaluation strategy; institutionalizing the
evaluation process so it becomes normal operating procedure;
and conducting cost-benefit studies.
NATIONAL WORKFORCE ASSISTANCE COLLABORATIVE ADVISORY GROUPS
NATIONAL WORKFORCE ASSISTANCE COLLABORATIVE PRODUCTS AND
SERVICES
1Evaluation of the outputs of a training program is called
summative evaluation. Evaluation of the training inputs and
processes, with the goal of improving the program, is called
formative evaluation.
2If a test measures what it is supposed to assess, it is said
to be valid. If the test yields consistent results, it is
reliable.