Level 4 Training Evaluations

level-4Knowing if your training program is having a positive effect on relevant KPIs, and are helping move your company toward its business goals, is a good thing.

Trainers do this by performing what’s known as a level 4 evaluation (in the traditional four-level Kirkpatrick training evaluation model).

In this article, we’re going to look more closely at evaluating a training program to see if it’s created the desired effect on the relevant KPI(s).

We’ll do that by looking more closely at the fourth of the four levels in Kirkpatrick’s evaluation model: the one that focuses on whether or not training has affected business goals.

Here we go. 

Before You Begin: Identify the Goals, KPIs, and Data Sources

Before you get to the point of evaluating your training program, you should:

  1. Find out which business goal(s) the training program supports.
  2. Find which KPI(s) track progress toward that goal and how to get the data.
  3. Design, develop, and deliver the training, keeping the business goal(s) in mind.

Then you’ll be ready to get into evaluation.

Evaluating A Training Program: Four Levels

The standard evaluation method is Kirkpatrick’s Four-Level Evaluation Model.

The Kirkpatrick evaluation model breaks evaluation down into four levels:

Evaluation Level 1: Reaction

In level 1, we’re getting the opinion of the learners who attended/completed the training. These are the post-training evaluation sheets (sometimes dismissively referred to as “smiley sheets”) that are handed out after training.

Evaluation Level 2: Learning

In level 2, we’re trying to figure out if the learners “learned.” This is typically measured with some form of assessment during or immediately after training. This might mean a paper-based test, an online quiz, or some form of skill demonstration that’s evaluated by the instructor.

Evaluation Level 3: Behavior

In level 3, we’re concerned with whether or not workers are actively applying what they learned when they return to the job. This is what we’re talking about when we talk about “transfer.”

Evaluation Level 4: Results

In level 4, we’re talking about the effect that the training program had in helping the company reach a business goal. Progress toward those goals is measured using KPIs. And that’s what this article is about.

workforce-training-mgmt--blue

How Often and When Will You Perform Level Four Evaluations?

Two questions that come up when thinking about level four training evaluations are:

  1. Will I do this for every training program?
  2. How long after the training program should I do this?

We’ll give some helpful information about each question in the sections below.

How Often Will You Perform Level 4 Training Evaluations?

You may wonder if every company performs all four levels of evaluation for every training program they create and deliver. The short answer is no. Nobody’s got time and money to do that all the time.

Below are some benchmarks to consider for evaluating at different levels. Check them out. (The information below includes a fifth level for “ROI,” which is the business effect put into monetary value–we’ll address this in a later blog post.)

From left to right, the table shows recommended targets for each level by the authors of a book on ROI; targets established by the GAO; real-life figures in public sector (federal, state, and local governments) as determined by a research study, and targets that Wachovia Bank has established for themselves.

evaluation table

Source: “Return on Investment: ROI Basics”  by Patricia Pulliam Phillips and Jack J. Phillips, ASTD/ATD Press, December, 2005, page 30. 

When Should the Level 4 Training Evaluation Take Place?

As explained above, once the training has been delivered, you can return and evaluate the change in the relevant KPI(s).

Which raises an interesting question. How long after training should you wait?

Let’s return to the book  Return on Investment: (ROI) Basics, already referenced above, for some expert opinion on that issue:

“Levels 3 and 4 data collection occurs sometimes after the new performance has had a chance to occur–the time in which new behaviors are becoming routine. You do not want to wait until the new behavior becomes inherent and participants forget where they learned these new behaviors. Typically, Level 3 data collection occurs three to six months after the program, depending on the program. Some programs, in which skills should be applied immediately upon conclusion of the program, should be measured earlier–anywhere from 30 days to two months after the program. Level 4 data can be trickier, however.

While the ROI calculation is an annual benefit, do not wait a year to collect the Level 4 data. Senior executives won’t wait; the problem will either go away, executives and senior managers will forget, or a decision will be made without the data. Collect the Level 4 measures either at the time of Level 3 data collection or soon after when impact has occurred. “

Source: “Return on Investment: ROI Basics”  by Patricia Pulliam Phillips and Jack J. Phillips, ASTD/ATD Press, December, 2005, page 70.

Level 4 Evaluation: Examples of Training Programs Affecting a KPI

Assuming your training program has been successful, it will help move your business toward a business goal. (Yes, your program can have no effect, or even a negative effect, but we’ll ignore that sad possibility for now.)

As a result, the KPI that measures progress toward the business goal will change.

In some cases, the KPI will go “up,” meaning progress has been made. In other cases, the KPI will go “down,” but this can still mean that progress has been made. It depends on the KPI you’re tracking.

Let’s look at two quick examples.

Level 4 Training Evaluation Example: Net Profit

One example of a commonly used KPI is net profit. It’s possible that this is the KPI or one of the KPIs you’ve chosen to track to determine if you training program had the desired influence.

In this case, you’d want to see the KPI go “up” after the training program was implemented, because when you’re talking about profits, going up is good.

net profit graph

And, as you can see above, in this case the net profits did go up after the training.

Level 4 Training Evaluation Example: Safety Incidents

Another example, this time from safety or EHS, would be workplace injuries or illnesses (sometimes referred to as “incidents”). Because safety incidents are bad, and the goal would be to have fewer, in this case you’d want to see the KPI go “down” after the training program.

safety incidents

And, as you can see above, safety incidents did go down after the training.

In both examples above, the data have been simplified, but you get the idea. In each case, you’ve got some strong evidence that the training program created the desired effect for the business goal. In the first example, net profits went up after the training program was implemented. In the second example, safety incidents went down after the training program was implemented. Both were successes–good job, training team!

workforce-training-mgmt--blue

Additional Points about Showing That a Training Program Had Desired Effect on a KPI

Savvy readers like yourself no doubt notice the sleight of hand above.

Yes, the KPI went in the desired direction (up or down) in the two examples above. And yes, the KPIs moved in the desired direction after the training program was implemented.

However, as we all know, correlation does not imply causation. Which is a fancy way of saying that even though the KPI went up in the first example and down in the second example, and the desired changes occurred after the training program was implemented, we don’t have enough evidence to prove that the training program is what caused the desired change.

That’s what you were thinking, right?

If so, congratulations. Because you’re right. You still need to do what’s called “isolating the effect of the training program.” This means controlling the effect that other variables might have had on the KPI so you can determine how much of the change is due to the training program. For example, in the first case, the rise of net profits after the training program might have been the result of a simple price change or a new advertising campaign. And, the decrease in safety incidents in the second example might have been the result of the installation of a bunch of new machine guards.

So we’re going to write another blog post about isolating the effect of the training program. Don’t let us forget–feel free to remind us. 🙂

[Post-publication edit: we’ve now completed the promised article about isolating the effects of your training program.]

In addition, we’re also going to write a second follow-up post to show you how to quantify that data and convert that data into financial terms (you know–dollars and sense) and create an ROI. That will make it much easier to communicate the value of your training program to others at your company.

So stay tuned for both.

In addition, you may find this related post that looks at providing graphic evidence that a training program has had a desired effect within a manufacturing training setting, providing examples based on safety, production efficiency, and quality, interesting.

Conclusion: Demonstrating that a Training Program Has Affected a Business KPI

You’ve now read an overview of how to perform a level four evaluation of your training program, showing that the training helped your company make progress toward a business goal as measured by a relevant KPI.

What are your own experiences? How often do you do level 3 and level 4 evaluations? Which KPIs are most relevant at your workplace? How often do you go the further step of isolating the effects of the training program, quantifying the change in the KPI, and converting that to dollar figures?

We’d love to hear your thoughts below.

manufacturing-training-guide

Jeffrey Dalto

Jeffrey Dalto

Jeffrey Dalto is an Instructional Designer and the Senior Learning & Development Specialist at Convergence Training. Jeff has worked in education/training for more than twenty years and in safety training for more than ten. You can follow Jeff at LinkedIn as well.

Leave a Reply

Your email address will not be published. Required fields are marked *