Evaluating EHS Training Effectiveness: ANSI Z490.1, Section 6

ansi-6 (Note: This article is based on the newly revised, 2016-version of Z490.1.)

Hello. Here’s another installment in our look at ANSI Z490.1, the standard that lists “Criteria for Accepted Practices in Safety, Health, and Environmental Training.”

This post focuses on best practices for evaluating EHS training. That means two things: evaluating how well your employees learned from their EHS training and, of course, evaluating the EHS training itself. Plus, it means using that evaluation information during continuous improvement efforts.

If you want to download our free 42-page Guide to Effective EHS Training, based on all of ANSI Z490.1, click that link you just whizzed past or scroll down to the bottom of this article and click the download button.

Otherwise, let’s get learning about evaluating EHS training.

Evaluating EHS Training

As you probably guessed, the purpose of evaluating EHS training is to see if it the training was effective. Are your employees “learning” from the training? Does the training lead to the desired change in their behaviors? Does one or more individual employee need additional help after the training? Do you need to modify the training?

Section 6 is divided into three parts:

  • Criteria for evaluating EHS training
  • Ways to evaluate EHS training
  • Continuous improvement efforts

Let’s look at each of the three below.

Criteria for Evaluating EHS Training

The standard provides some general criteria for evaluating EHS training, including the following.

Begin Planning for Evaluation Back During Development

Before you deliver the training to employees, you’ve got to design it and develop it (that’s what’s covered in Section 4 and our article about that section, How to Develop Effective EHS Training).

Even that early, you should be thinking about your post-training evaluation of the learners and evaluation of the training materials themselves. Outcomes that can be evaluated include (but are not limited to):

  • Does the employee possess a particular desired knowledge, skill, ability, and/or attitude after training?
  • Does the trainer to effectively help employees acquire the desired knowledge, skills, abilities, and/or attitudes during the training?
  • Are the employees satisfied with the training experience?
  • Is the training contributing to organizational goals?

The Post-Training Employee Evaluation: Can They Satisfy/Meet the Learning Objectives?

One thing to evaluate is whether or not each employee in the training successfully satisfies the each learning objective of the training.

We explained what learning objectives are in our article on ANSI Z490.1 Section 4, but you can also click here or here to read more about learning objectives.

Identification of Employees Being Assessed/Evaluated

Every employee being tested must be properly identified. Be sure to create a mechanism for this.

Pre-Testing and the Possibility of “Testing Out”

During EHS training design and development, give thought to the possibility of allowing employees to take some form of pre-test and “test out” of the training.

Typically, if you DO allow for the possibility of testing out, the employee would do this by successfully completing the assessment that’s provided to employees after the training.

Take care to check regulatory guidelines to make sure testing-out is acceptable.

Criteria for Successful Completion

During the design and development phase, you’ll also have to determine the standard an employee must meet in order to complete the post-training evaluation successfully.

The standard for successful completion that you choose will vary on a number of things, including the type of evaluation you use. However, some possibilities may include:

  • If the employee has to take a test, successful completion may mean answering a specific number of questions correctly or answering a specific percentage of questions correctly
  • If the employee has to demonstrate that he or she can perform a task correctly, successful completion may require performing all steps in a specific order and manner
  • If the employee must complete a project or some sort, successful completion may require that the employee include all key elements in the project

Evaluation Must Comply With All Industry Standards and Regulations

Also remember that the evaluation must comply with all industry standards and regulations. For example, regulations often specify a minimum acceptable level of training (although it’s always OK to exceed those requirements).

EHS-training-course-library

Reliability and Validity

When you create an assessment to evaluate if the employees can meet the learning objectives, it’s important to know the evaluation must be “reliable” and a “valid measure.” What does this mean? Here’s what the standard says:

  • Reliable: Gives consistent results over time
  • Valid measure: Reflects the knowledge, skills, abilities, or attitudes specified in the learning objective

You can read this article for more about reliability and validity.

Providing Results to Employees

Another thing to think of in advance is to create a manner in each employee who took the training will be provided with the results of the evaluation (whether it the “scored” test or the observed skill demonstration).

Trainees can use this as part of the “feedback” to help guide them as they look for additional information or practice and also to help them create a plan for future training.

Periodic Re-Evaluation of Employees

The training program should include some form of periodic evaluation of the trainee’s ability to satisfy the learning objectives. This helps to evaluate the training as a whole as well as the need for retraining or refresher training.

Training Evaluation Approaches

Now that we’ve covered some general criteria for training evaluation, let’s look more closely at some approaches or methods for evaluating training.

Section 6 of the standard suggests a method that’s based on what is commonly known as Kirkpatrick’s Four Levels of Evaluation.

In this method, training is evaluated at as many as four different levels to determine if it’s effective. Those four levels are:

Next, let’s turn our attention to the type of evaluation to use. The evaluation should include one, and may include all, of the following approaches:

Level 1: Employee Reaction

After the training is complete, have the employees give their opinion of the training. This can be delivered as a survey. Be sure to include space for written comments, too.

Survey questions can cover topics such as the trainer’s presentation skills, the suitability of the training environment, the pace and difficulty of the training, and the usefulness of the training material.

These surveys can help the trainees assess their own level of learning. In addition, they can be used by trainers and training providers to assess and improve the training materials and training delivery.

A good source of information about creating useful level-1 survey sheets, often called “smile sheets,” is the book Performance-Focused Smile Sheets: A Radical Rethinking of a Dangerous Art Form by Dr. Will Thalheimer.

Level 2: Assessment of Employee’s Knowledge, Skills, Abilities, and Attitudes

This is an assessment, during or immediately after training, to determine if the employees can satisfy the learning objectives.

This assessment may take several different forms, including:

  • A written or online test
  • A performance assessment in which the worker demonstrates a skill while being observed; this can happen in a simulated work environment or the real work environment
  • An oral examination
  • The completion of an assigned project

You may find these articles helpful while planning your own post-training assessment:

Level 3: Observation of On-the-Job Performance

Because training is intended to change how people perform on the job, it’s important to observe the workers in the field to see if they can demonstrate the skills/abilities/knowledge/attitude from the training while actually working.

This can be done after training, but it can also be done both before and after training to get a better sense of any improvement the training may have caused.

On-the-job observation can come from direct observation, but also from information collected by supervisors, coworkers, or customers, or also from indirect measures in product records or safety reports.

If observation detects a gap between the actual performance and the desired performance, attempt to determine the cause for that gap. In some cases, this can be tracked back to problems with the training and/or training delivery, but in other cases, other factors that are un-related to training may exist (this article on identifying and closing skill gaps explains this in more detail and gives a demonstrates a method for investigating this issue).

Level 4: Affect of Training on Organization/Business Goals

Finally, you may try to evaluate the effect that the training had on goals of the organization or business.

This can include tracking things like:

  • An increase in observed safe behaviors
  • An increase in preventive measures and controls implemented
  • A reduction in near misses, injuries, and/or illnesses
  • A reduction in insurance claims
  • Improved environment compliance
  • A return on ROI of training

You may have noticed that to do this, you’ll have to select particular key performance indicators (KPIs) of the business that you’ll track over time. The following articles provide more information on this topic:

Continuous Improvement of the EHS Training Program

As we mentioned earlier, the evaluation phase of EHS training has two primary purposes:

  • Evaluating each employee to see if he or she can satisfy the learning objectives of the training and has acquired the desired knowledge, skills, abilities, and/or attitudes
  • Evaluating the EHS training program/trainer/material itself to determine if it’s effective and how it can be improved

That second bullet, above, is what “continuous improvement” is all about.

Use all information from training evaluation to improve court content, delivery methods, materials, and learning environment.

EHS training developers and trainers should use the results of the learner evaluations and all other training evaluations to periodically review the effectiveness of the training materials and training presentation themselves. Evaluation should include course content, training delivery methods, additional training materials, trainer performance, learning environment, and more.

Remember to also use information from incident investigations, job-site observations, safety audits, and inspection data to make improvements to the training as well.

Conclusion: The Importance of Evaluating EHS Training

No doubt you recognized the importance of evaluation the effectiveness of your EHS training and the learning of your employees even before you read this article. With luck, this article has given you some best practices you can use to evaluation your EHS training more effectively.

Of course, we encourage you to go get the ANSI Z490.1 standard as well.

The Rest of ANSI Z490.1

Here are links that lead to the rest of the articles in this series:

Don’t forget to download our free guide, below.

OEE-guide-btn

Effective EHS Training: A Step-by-Step Guide

Learn how to design, create, deliver, and evaluate effective EHS training by following these best practices with our free step-by-step guide.

Download Free Guide

OEE-guide-btn

 

Jeffrey Dalto

Jeffrey Dalto

Jeffrey Dalto is an Instructional Designer and the Senior Learning & Development Specialist at Convergence Training. Jeff has worked in education/training for more than twenty years and in safety training for more than ten. You can follow Jeff at LinkedIn as well.

Leave a Reply

Your email address will not be published. Required fields are marked *