If you’re in training, you’re probably familiar with the sheets that trainers pass out to learners after a training session, asking the learners to evaluate the training session and the trainer.
These are known by a variety of names. Maybe you call them training-evaluation forms, or student-response forms, or trainee-reaction forms. But they’re also commonly–maybe most commonly–known as smile sheets.
Why smile sheets? Because it’s common for the learners attending training to give the training/trainer high scores that make everyone smile. But the common assumption is that the trainees do that politely, kindly, quickly, uncritically, and without giving any great thought. And so the term smile sheet is generally used somewhat dismissively, with the assumption that the information they contain doesn’t really provide a lot of value.
And yet, quite a few trainers continue to use smile sheets, and many of those trainers do nothing to improve them. Maybe they’ve never even thought of improving their smile sheets. It’s all become a bit of a habit to them, one they don’t think about because there’s so much else to think about, worry about, and to do.
I recently read a very good book called Performance-Focused Smile Sheets: A Radical Rethinking of a Dangerous Art Form by Dr. Will Thalheimer. The book explains some of the common problems with smile sheets, but also gives some very helpful tips to help make them better. We definitely suggest that you buy and read the book, and we’ve included a bunch of information to help you do that at the bottom of this article.
But for now, let’s look at some of the general points Thalheimer makes in his book and see what we can learn from them.
Some Basics on Smile Sheets
Let’s take a look at a few basic issues about smile sheets and their use, including:
- Who uses them and who doesn’t
- What’s the point?
- Some thoughts about their place within Kirkpatrick’s four levels of evaluation
Who Uses Smile Sheets?
In my own unscientific poll, you can break trainers down into five groups (mea culpa: that’s bad that I’m using an “unscientific poll,” because we should be using data-driven, evidence-based methods, but the exact accuracy of this breakdown isn’t a critical issue for our purposes today, so we’ll go with it):
- Trainers who have never heard of nor used smile sheets
- Trainers who have heard of smile sheets, but don’t use them primarily out of laziness
- Trainers who have heard of smile sheets, but don’t use them because they think they bring no value
- Trainers who use smile sheets, but use traditional smile sheets that bring little value
- Trainers who use smile sheets, and who have somehow put shoulder to rock and, largely on their own, and improved the smile sheets they use so they deliver real value
We think this article will be of interest to those different groups for different reasons. Here’s our thinking:
|Type of Trainer||Why You Might Care|
|Never even heard of ’em||We are all interested in knowing about stuff from our field that we don’t yet know and that might possibly help us, no?|
|Heard of ’em, too lazy to use ’em||Hey, we get it. Sometimes we’re all tempted to be a bit lazy. But if you read on, you may find that these can deliver enough value to make them worth the effort.|
|Heard of ’em, don’t use ’em ‘cuz they’re no good||Well, good point. That’s largely been true up to now, although there are exceptions. But our goal here is to show you that you can make really good ones.|
|Heard of ’em, use ’em, but use the ones that provide little or no value||OK, so you’re making them. Maybe even because you think you have to. But since you’re already making them, we bet you won’t resist making better ones that provide more value.|
|Heard of ’em, use ’em, already write great ones||Great! Well, maybe you’re doing fine on your own, but we bet that if you’ve already gone to the effort of figuring out how to make good ones, you’re also interested in learning more. Right?|
What’s the Point of Using Smile Sheets?
So you may be thinking what’s the whole point of using smile sheets anyway? Seems like a good place to start the conversation.
There are plenty of good reasons to write smile sheets. Thalheimer gives a list of nine reasons (which he explains he himself modified from an earlier list created by leaning measurement expert Rob Brinkerhoff). Two that are kind of clever, that raise legitimate points, and that you might not think about are:
“Upholding the spirit of common courtesy by giving learners a chance for feedback” (1)
“Enabling learner frustrations to be vented-to limit damage from negative back–channel communications.” (2)
But the real reason for using smile sheets, to collapse a few of Thalheimer’s points into a single point, is to identify weaknesses in current training and use that information to make future training better.
Here, I want to quote Thailheimer at length, because he writes about this passionately and eloquently in his Preface:
“I took this path [studying learning & development] because I believed strongly–and still believe–that learning is a noble cause. It is learning that has enabled human civilization and growth. It is learning that enables individuals to excel and thrive. It is learning that holds the promise of the future.
If learning is so important and our task is such a noble one, don’t we, as learning professionals, have an almost sacred responsibility to do our jobs well?
The way I see it, there are two main lynchpins to our performance. First, scientific research must guide our starting assumptions. Second, we must use good learning measurement to get valid feedback so that we can refine our understandings, improve our learning design, and live up to our promise-so that we can maximize the benefit of learning…
While smile sheets should never be the only way we get feedback on learning, by improving them, we can get significantly better information about how we’re doing. With better information, we can create virtuous cycles of continuous improvement. We can build more effective learning interventions and meet our obligations as learning professionals.”(3)
Kudos! Well said, right?
If we never try to evaluate the effectiveness of our training, we can’t use that data to see if our training is working well or not. Which means if our training isn’t working, we may not know that, and we won’t have helpful information to make it better. And it also means we can’t apply the same lessons to other training we may create in the future. So not evaluating the effectiveness of training is a loss/loss.
Consider that phrase Thalheimer uses–“virtuous cycles of continuous improvement.” Maybe it’s just the third cup of coffee in me talking, but I love that as a professional goal in any discipline. You’d think our friends in lean manufacturing, with their emphasis on kaizen, would be big fans of Thalheimer’s efforts.
Smile Sheets and Kirkpatrick’s Four Levels of Evaluation
If you’re familiar with Kirkpatrick’s Four Levels of Training Evaluation, you may know that smile sheets are level one.
If you’re not familiar, and you want the exenteded introduction, check out that link above.
If you are familiar, or just want a quick review, check out the bullets and image below.
Kirkpatrick’s Four Levels of Training Evaluation
- Level 1-Reaction: Smile sheets–surveys handed out to the learners, online evaluations, etc.
- Level 2-Learning: Role-playing scenarios during training, focus groups, case studies, tests, etc.
- Level 3-Behavior: Observation of on-the-job behaviors and other evaluation metrics.
- Level 4-Results: Direct measurement of business goal–cost, revenue, production, etc.
|Type of Trainer||Thoughts on Level 1 (Smile Sheets) and Relation to Other Levels|
|Believe that positive learner reactions lead to positive learning, behavioral, and business outcomes
(Note: Here, “positive learner reactions” means learners said they liked the training or that they learned a lot from the training).
|A positive learner reaction is going to lead to positive learning, behavioral, and business outcomes. A negative reaction won’t.|
|Believe that positive learner reactions don’t necessarily lead to positive learning, behavioral, and business outcomes
(Note: Here, “positive learner reactions” means learners said they liked the training or that they learned a lot from the training).
|A positive learner reaction may or may not lead to positive learning, behavioral, and business outcomes. The same is true with a negative learner reaction.|
The interesting distinction here is whether or not a positive level 1 (smile sheet) evaluation is necessary for effective learning and behavioral change, or even if there’s a correlation between having the learner say they “liked” the training or say that the training was “effective” and the reality of whether or not the training prepared them to perform well on a test and/or if the training will lead to desired on-the-job behaviors and movement toward desired business goals.
We’re going to discuss that in more detail in the very next section, but before you move on, give this question some thought yourself. Is it more likely that employees learned during training if they said they “liked” the training or that the training was “good?” And what if employees rated one training activity (or one trainer) lower than another—are those employees less likely to learn, to perform desired behaviors on the job, or to contribute toward the business reaching business goals?
What do you think?
What’s Wrong With Traditional Smile Sheets?
There are quite a few problems with “traditional” smile sheets. Thalheimer goes over them in detail in his book(4), and we’ll briefly cover a few of the key points below.
Probably the big bomb-shell Thalheimer delivers is that, according to two meta-aanalyses of studies on smile sheets (level 1 evaluations) and their correlation to test results (level 2 evaluations), there’s ALMOST NO CORRELATION with learning results.
Again, a little quote from the book:
“They found that smile sheets were basically uncorrelated with learning results! To be specific, they were minutely correlated with learning-test results at a correlation of 0.09. You will remember from your statistics that correlations go from -1 to 1. Correlations between -0.30 and 0.30 are considered weak correlations. Having a correlation of 0.09 is practically no correlation at all. It would be like correlating the household level of peanut butter use with the household level of television use.” (5)
So that’s a biggie right there.
Some of the other problems that Thalheimer notes with traditional smile sheets are:
- We’re asking the wrong kind of questions
- We’re giving our employees the wrong kind of answer options
- We’re getting feedback that doesn’t tell us how effective the training was
- We’re getting feedback that isn’t actionable
- We’re translating the answer options we get into a numerical score
We’ll look at each of these issues in a little more detail in the section below.
Improving Traditional Smile Sheets: Some Tips on How to Do It
Let’s look at some of the biggest problems with smile sheets and get some ideas of how we can improve.
We’re Asking the Wrong Kinds of Questions
One of the big problems is that we’re asking the wrong kinds of questions.
The kinds of questions that we often ask relate to things like:
- Did you like the training?
- Did you learn from the training?
But studies show that having a high score for “liked the training” and/or “learned from the training” doesn’t mean the training was actually effective. In short, that means that the people who attend a training aren’t good judges of whether or not they learned.
What we should be do is asking if the learners feel that the training prepared them to apply the skills explained during the training when they return to the job.
Breaking that down a touch further, Thalheimer recommends questions that get at the four following issues:
- Do the learners understand?
- Will the learners remember?
- Are they motivated to remember?
- Are there after-training supports in place? (6)
Go ahead and pick up a copy of the book to learn more about this. Thalheimer goes into great detail on these issues, explains their importance, and includes a lot of sample questions too.
We’re Giving Employees/Learners the Wrong Kind of Answer Options
Many smile sheet questions ask the learner a question and force them to answer by choosing one of five options, which typically include options like Strongly Disagree; Somewhat Disagree; Neither Disagree Nor Agree; Somewhat Agree; and Strongly Agree. Or the options are similar to that. This is commonly known as a Likert scale.
There are at least four problems with these kind of answer options:
- They’re vague and it’s not clear where the “boundary” between one and the other is. When does Strongly Disagree shade into Somewhat Disagree? What’s the difference between Somewhat Agree and Strongly Agree?
- They may mean different things to different people. Your Somewhat Agree may be the same as my Neither Agree Nor Disagree, or even our coworker’s Strongly Agree.
- They’re not related to learning effectiveness and/or job preparedness. Even if the question is phrased well, so that it’s targeted at valuable information about training effectiveness and later, on-the-job performance, these answer options don’t give the learners a chance to correlate the training to whether or not they’ve learned the material and are prepared to apply it on the job.
- They don’t give trainers actionable information. So you gather all the responses, add up all the numbers, divide by the number of respondents, and you learn that for question 1, you’ve got an average score of 4.2 (out of 5). Is that good or bad? Where’s the cut-off between good and bad, exactly? How can that 4.2 score help you in your “virtuous cycle of continuous improvement” of training materials? If you do want to improve something in the training, what should you improve?
What’s a better way to write these smile sheet answer options?
The screen grab below (taken from a recent webinar we hosted on Effective EHS Training) shows some better answer options (and a better question, relating to the last section, too). You can see these answer options are:
- Much less vague
- Related to ability to transfer training to later on-the-job performance
- Written to provide feedback that gives the trainer insight into whether or not the training was effective
- Written to provide feedback that provides valuable information to the trainer who wants to improve the training
We’re Getting Smile Sheet Feedback that Doesn’t Tell Us How Effective the Training Was
For the reasons we explained above, plus others, the feedback we’re getting from our smile sheets doesn’t tell us if our training was effective or if it wasn’t. This is one of two things that Thalheimer says a smile sheet should do.(7)
We’re Getting Smile Sheet Feedback that Isn’t Actionable
And, also for many of the reasons we’ve explained above, we’re getting feedback from our smile sheets that doesn’t let us–as training developers, trainers, training materials, etc.–act to appropriately follow-up on the information from the smile sheet in order to improve our training. Remember those “virtuous cycles of continuous improvement?”
We’re Translating Smile Sheet Feedback Into Numbers and Averages
Not only are we asking the wrong kinds of questions, and not only are we giving learners the wrong kind of answer options, and not only are we getting feedback that doesn’t tell us how effective the training was and that isn’t actionable, but because we often use a five-point Likert scale as answer options, we often translate those answers into a number–like 3.7 or 4.2 out of 5.
What’s a 4.2, again? Is that good or bad? What should I do if I get a 4.2?
Can you reminder me again why we thought it would be a good idea to take poor information and make it one step more abstract?
More About Performance-Focused Smile Sheets: A Radical Rethinking of a Dangerous Art Form by Dr. Will Thalheimer
So we encourage you to run, not walk, to go buy yourself a copy of Performance-Focused Smile Sheets: A Radical Rethinking of a Dangerous Art Form by Will Thalheimer.
We haven’t addressed this in the article yet, but here’s more that we can say about the book:
- It’s (admirably) short and gets to the point
- It’s based in research and is evidence-based
- It’s funny; Thalheimer takes his subject seriously but not himself, and you’ll have some good laughs along the way
In addition, Thalheimer is a positive influence on social media, and fights the good fight for advancing good, data-driven, evidence-based training learning practices.
Here’s some additional information about Dr. Thalheimer and his book:
- SmileSheets.com: This is where you can buy the book. As Thalheimer explains, if you buy it here instead of elsewhere, he gets a bigger slice of the cut and can therefore afford to spend more time researching stuff that helps us all.
- Will At Work Learning: Thalheimer’s blog. Always interesting.
- Podcast with Will Thalheimer and Connie Malamed: Listen in to a 24-minute podcast during which Thalheimer discusses his book and the topic of smile sheets with Connie Malamed (one of my favorite instructional design bloggers).
Hope you enjoyed this article, good luck with your smile sheets, and let us know your thoughts in the comments section.
And hey, why not download our free guide to Effective Manufacturing Training?
Effective Manufacturing Training Guide: Manufacturing Training from Scratch
Discover how you can improve your Manufacturing training program by following these best practices with our free step-by-step guide.
Book Cover and Notes
And finally, here’s the book itself, and your proverbial opportunity to judge a book by its cover. Notes from this article are below the book cover.
- Thalheimer, Chapter 1 (What Are Smile Sheets For?), Location 257. Apologies to those of you who might by this as a physical book–I bought it as an e-book and it doesn’t display page numbers 🙂
- Thalheimer, Chapter 1 (What Are Smile Sheets For?), Location 257.
- Thalheimer, Preface, Location 109-119.
- Thalheimer, Introduction, Location 133-143 (he’s got a list of 10 problems there).
- Thalheimer, Chapter 2 (Your Smile Sheets Suck!), Location 325. The two meta-analyses he references are one from 1997 (Alliger, Tannenbaum, Bennett, Traver, and Shotland) that included 34 separate studies, and a second from 2008 (Sitzmann, Brown, Casper, Ely, and Zimmerman) that covered 136 separate studies. Both meta-analyses found exactly the same weak correlation-0.09 (what are the odds of that?).
- Thalheimer, Chapter 3 (Smile Sheets Should Predict Training Effectiveness), Location 808.