Writing Better Tests for Job Training: The Issues of Reliability and Validity

writing-test-questions-for-online-training-activitiesIt’s often, if not always, a good idea to provide some form of test or assessment after providing job training to employees.

In some cases, this may be a written test scored in a pass/fail manner, and in others, it may be a performance test that requires the workers to demonstrate a skill or the ability to perform a procedure in a satisfactory manner.

In either case, it’s important for that test to be a good one. By that we mean that it provides you withuseful, actionable information about whether or not the employee has truly benefited from the training and is ready and able to successfully apply the new information or perform the new skill on the job.

There are a number of characteristics that “good tests” like this share. Learning & development experts know the two that we’ll talk about in this article as validity and reliability.

Valid Tests

A valid test is an accurate test of the employee’s knowledge, skill, or ability.

Let’s look at an example: say you’ve got a team of machine operators, and you give them each a test to see if they can operate the machine. Half pass the test, and half fail.

Then let’s say you take those workers out onto the production floor and let them run the machine.

If your test was valid, the half who passed the test will be able to run the machine, and the half who failed the test won’t be able to run the machine.

On the other hand, if your test is not valid, some of the half who passed the test won’t be able to run the machine, and/or some of those who failed the test will be able to run the machine.

Tips for Creating Valid Tests

Now that you know what a valid test is and see why it’s an important concept, here are a few tips for creating valid tests:

  • Make sure your test matches your learning objective. For example, if you want someone to be able to operate a machine, don’t create a test that asks them to explain how to operate a machine. Instead, assess their ability to actually operate it. Or, don’t test their ability to operate the machine while it’s running much more slowly than it will when they need to operate it on the job. Or, don’t test them in an environment with fewer distractions than they’ll have to deal with on the job. Instead, create a test that evaluates their ability to operate the machine at normal work speed and with normal work distractions.
  • Match the difficulty of the test with the difficulty of the real-world task. This is similar to the point above, but bears repeating. Don’t make the test easier than the real-life task, or else you won’t know if your employees can perform the task on the job. And likewise, don’t make it harder, either–don’t test someone’s ability to run a marathon if they simply have to job 50 yards (that’s an analogy, but you get the point).
  • Ask real-world experts (so-called subject matter experts) for their input in creating the test to match real-world expectations and experiences.

Reliable Tests

Now that we’ve got validity covered, let’s look at our second criteria: reliability.

If your test is reliable, it will tell you every time if an employee can satisfy the learning objective, and it will likewise tell you every time if an employee can’t meet the learning objective.

In short, when everything else is the same, a reliable test will give the same results every time. A reliable test is a consistent measurement. An unreliable test is not consistent.

For a visual analogy, think of a dart board with four darts all thrown in the same area of the board. For this example, we’ll say that all the darts are thrown very near the bulls-eye at the center of the dartboard. Those four darts represent the passing score on a test taken by the same employee four different times. This is a reliable test. The employee gets the same or very similar results each time.

For a second analogy, think of that same dart board with four darts again. The darts are all closely grouped again. But this time, all the darts are off to the side of the board (near what we’d call 3:00 on the face of a clock). Those four darts represent the failing score of the same employee who’s taken the test four times in a row. This is a reliable test again, because the same employee got the same results each time (even though he/she has failed).

And for a third and final visual analogy, think of a dart board with four darts spread all over the dart board. Those darts represent the different scores and results when one worker takes a test four times in a row. This is not a reliable test, because the same employee gets wildly different results each time.

So, your goal is to create tests that led to reliable results–meaning results that consistently show a passing employee has passed and a failing employee has failed. And, on the flip-side, your goal is to NOT create unreliable tests.

Tips for Creating Reliable Tests

Now that you know what a reliable test is, let’s give you some tips for creating them:

  • Create tests that are at the correct difficulty level–match the test to the necessary job performance
  • Create tests with enough test items so you can get a true understanding of whether or not your employees can satisfy the objective. In a perfect world, you’ll have several test items for each objective (this reduces the chances of something random skewing the results).
  • Create well-crafted, well-thought-out, well-designed, well-written tests that aren’t vague, misleading, ambiguous, poorly written, or easy to guess. Review the items and have others review them for you before trying them for real with your employees.
  • Don’t create tests that are difficult to answer or difficult to answer just because of the user-interface. In other words, don’t get too cute or be so sloppy that your employees can’t figure out how to answer.
  • Create tests with an objective scoring basis.
  • Don’t leave scoring up to the whims of one evaluator, or the different whims of multiple evaporators. Create a checklist or guide for evaluators so they all know what to look for and how to assess worker performance.

Good Tests Are Both Reliable and Valid

You may have gathered this already, but we wanted to call it out directly: A good test needs to be both reliable and valid.

This information is important if you’re having a training provider create tests for you.

Likewise, it’s important if you’re using your own learning management system (LMS) to create your own assessments, using tools like the online quizmaker shown below.

QuizMaker Quiz Question Creation

LMSs such as the Convergence Enterprise LMS allow you to make your own online quizzes and other online learning activities.

Conclusion: Good Tests for Good Workforce Training

Hopefully understanding the concepts of reliability and validity will help when you evaluate tests or create and evaluate your own tests.

Search for the word “test” for more helpful articles on related testing topics. Or, let us know if you’ve got some questions about testing (there’s a chat feature down in the bottom-right corner) or drop a comment in the Comments section below.

Creating Learning Objectives Btn

How to Write Learning Objectives

All the basics about writing learning objectives for training materials.

Download Free Guide

Creating Learning Objectives Btn
Jeffrey Dalto

Jeffrey Dalto

Jeffrey Dalto is an Instructional Designer and the Senior Learning & Development Specialist at Convergence Training. Jeff has worked in education/training for more than twenty years and in safety training for more than ten. You can follow Jeff at LinkedIn as well.

Leave a Reply

Your email address will not be published. Required fields are marked *