Common Biases In Level 1 Learning Surveys
In workplace learning, L&D's Level 1 evaluation, often known as "reaction" or "smile sheets," is one of the most common tools for measuring success. Satisfaction numbers and NPS scores can be obtained easily through an automated LMS survey. And the numbers look good, so we did our job! Right?
This article does not focus on whether smile sheet results are good indicators of application and impact on the job (hint: mostly not) but rather explores the intricacies of writing reliable, valuable, and practical Level 1 surveys. However, if you're interested in why NPS may not be the best metrics for learning, look at this Net Promoter Scores and Level One evaluations article exploring construct validity ("Are you measuring what you think you're measuring?") and predictive validity ("Is it predicting some desired behavior?") in the context of learning.
Tip 1: Start With The Why!
Why are you doing the learning survey? This is not a rhetorical question. For real: what is your goal with the survey? Do you need a pat on the back for doing well? Do you want to validate or reject your hypothesis on what works? Do you just need to raise the response rate? Do you want to monitor course or program performance only for big disasters? Are you willing to take any actions based on your data? Are you reporting on what happened or investigating why it happened? Are you providing predictive guidance on what might happen?
- No right or wrong answers. Just answers.
There are no right or wrong answers, but you need to be very clear about the intent of the survey before you design the instrument.
Who's The Audience For The Survey?
One of the misconceptions I've seen in the industry is that the Level 1 surveys are for learning designers and facilitators. And you wonder why the response rate is low? Are you telling employees to work for you (as in creating data for you) on top of completing some course or program while they're also busy doing their jobs? What's in it for them? Imagine someone filling out these forms, including open-text responses, for months or years and seeing no change. Not. One. Thing. Different. Or maybe different, but they would never know it was based on feedback. What's the point of providing feedback for them?
If you want to improve your response rate, you can make it mandatory (I strongly discourage doing that), or you can make your audience see the value of providing feedback. How would you do that?
Think of the surveys as a dialogue rather than data collection.
People are interested in whether their opinions match others. People are interested in the impact their opinions make. People do what leadership considers valuable and a priority. Share lessons learned from surveys with leaders. More about this later, because the data insights you gain from the traditional smile sheets are often at the bottom of the interest list of business leaders.
Tip 2: Mitigate Common Biases
I used to say "avoid" common biases, but I've learned that words matter. When learning professionals attempt to avoid these biases in their surveys and don't succeed, they may return to their old ways. It's all or nothing, right? Start small, think big. Progress over perfection all the time!
Common Pitfalls In Survey Design And Implementation
- Survivorship bias
It is a type of selection bias where only select users (those who survived the selection process) will be heard, therefore skewing the data. -
- For instance, are you sending surveys to only those who completed the course or program? Wouldn't you like to know why others dropped out?
- Ambiguous questions
One of the most frequent issues in survey design is ambiguity. Questions that are too broad or vague can lead to inconsistent responses. Remember, participants do not read your mind. They read your text only. Their interpretation of the words in a question may be different than intended. For instance: -
- Problem: "How satisfied are you with the content?"
- Reason: What is content? When I asked this question on LinkedIn, I got answers such as what's included in the course (topics), what's on the screen as text, the whole learning experience, etc. If your audience can easily misinterpret the question, how do you interpret their answers?
- Leading questions
Questions that lead respondents towards a particular answer can skew the results. This is also true for statements when you ask for the level of agreement. For example: -
- Problem: "How beneficial was the highly informative training session?"
- Reason: You're leading the witness by priming them with "highly informative"!
- Double-barreled questions
These questions ask about two different things simultaneously, confusing respondents. These questions often indicate a lack of clear definition for each component. For instance: -
- Problem: "Was the training engaging and relevant?" or "How would you rate your motivation and engagement after the training?"
- Problem: You can't be sure what participants' answers mean. They may interpret them as either of the two components or both. Something might be engaging but not relevant, or provide plenty of knowledge but no skills.
- Response biases
This includes tendencies like acquiescence bias, where respondents may agree with statements regardless of their true feelings, and social desirability bias, where they answer in a way they believe is more socially acceptable. -
- Mix it up: People have the tendency to agree with your positive statements. One way to address that is to introduce a negatively phrased statement or question. However, use it sparingly, preferably early on in the survey. This can make respondents pay more attention to survey questions throughout.
- Some of the biases are specific to the Likert scale question type, such as selecting extreme values or selecting neutral values all the time. Provide an "I don't know" or "Not applicable" answer to avoid skewing your data towards the neutral position.
- Inadequate response options
Providing a limited range of responses can restrict the data's usefulness, or may result in incorrect insights if used as the only data point for decision-making. For instance: -
- Problem: "Did you find the training useful? (Yes/No)"
- Reason: Not actionable. If they say "yes", then are we satisfied with our outcome? Wouldn't it matter how useful it was? If they say "no", then what? Do we abandon the training? Again, these questions should be used along with other questions. However, use them sparingly because the longer the survey, the less likely your audience will be to complete it.
- Likert scale dilemma
We love the Likert scale because it produces a number. We can compare and contrast the metrics. However, be aware of the "side effects" of the Likert scale. For example, "Fowler (1995) also noted that respondents are also more likely to use rankings on the left side of a continuum, regardless of whether the continuum is decreasing or increasing from left to right." -
- Another Likert scale issue is labeling options with words (strongly agree, agree, etc.). Because every label has different words, it is difficult for the respondent to treat them as a continuum. The distance between strongly disagree and disagree may be different from the distance between disagree and agree. If you need to use the Likert scale, label the ends of the scale only. Well-designed questions will produce a normal distribution.
Tip 3: Learning Survey Structure
Bias For Topics
People tend to respond similarly to questions they think relate to each other. If you have questions grouped in topics, mix up the order of questions, or at a minimum, do not label or indicate questions as part of a group [1]. Similar types of questions on a page (especially when there are many of them on a scrolling page) can cause "survey fatigue." Mix up the types and structure.
In the next article, we'll explore ways of making your Level 1 surveys more actionable, learn why sampling can be misleading, and try some alternative, experiential questions about behavior change.