As you know, there are many new technologies available now or coming soon that may influence learning and development in different ways. So many that it’s a little hard to keep track of them all, not to mention have an understanding of what they are and just exactly what influences they may have on workforce learning and development.
And of course, as often happens when a new technology comes about, it’s also easy to get a little over-excited and think this is the long-awaited, massive game-changer that will completely change learning and development forever, bringing with it all the solutions to all the problems and challenges we face.
To help us get a better understanding of all these new technologies, to see how we might use them in learning and development, and to see which may have some type of “disruptive” effect in workforce training, we’ve asked Dr. Stella Lee, a learning professional with a ton of experience studying technologies for workforce L&D, to share some of what she knows with us.
Before we begin, two quick points: First, we’d like to thank Dr. Lee for sharing her time, knowledge, and experiences with us. And second, know you can watch the recorded video of this discussion immediately below or click the MORE button to read the transcript I typed up for ya 🙂
And with that said, let’s start learning about disruptive technologies and their use in learning and development from Dr. Lee.
Disruptive Technologies in Learning and Development: A Talk with Dr. Stella Lee
Convergence Training: Hi there, everybody, and welcome. This is Jeff Dalto of Convergence Training and we’ve got another one of our semi-regular webcast/podcast/interviews today.
Today we’re really excited–we’re over in the world of learning & development (L&D) and we’re talking with Dr. Stella Lee. Stella is the owner of a learning consulting firm called Paradox Learning. Stella is modest and told me in advance that she didn’t want to begin by explaining her own experiences, but briefly I’ll tell you that I’ve been aware of Stella for several years on LinkedIn, I read all of her stuff and I’ve learned a ton in doing so about today’s topic, which is disruptive technologies in learning, including what they are, what are some of the promises, what are some of the drawbacks, and some cautions and opportunities.
Stella has been in L&D for 20 years. She started in academia, from there she to the government and then to the private sector, and for the last five years she’s been working as a learning consultant at Paradox Learning, her own company. And I’ve seen Stella talking at any number of impressive places on any number of impressive topics–especially in the last year–Google, Amazon and elsewhere. She’s a really great source of information, so we’re happy to have Stella with us and with that, Stella, hi and how are you today?
Dr. Lee: Hi Jeff, it’s great to be here, thanks for having me.
What Are Disruptive Technologies?
Convergence Training: Oh, our pleasure. Well, I guess to get right to it, if we asked you here to talk about disruptive technologies in learning, maybe you could start by telling us what this means–disruptive technology, what the heck is that?
Dr. Lee: (Laughs). I know, there’s a little bit of confusion there. The way I look at it, the way the definition goes, is anything that forces you to rethink or transfer the way you do business.
Think about Blockbuster, the video store. So what disrupted Blockbuster? Netflix. Or more so the technology behind it, which is on-demand streaming (video). And so that forces the industry to change, either to rethink how they would do their business or it would force you to go out of business in the case of Blockbuster.
So that’s what disruptive technology does, fundamentally changing how you go about conducting your business as usual.
Convergence Training: Great. So, since you talked about Blockbuster, the one brick-and-mortar Blockbuster still left is near my current hometown. I’m outside of Portland, Oregon and Bend, Oregon has the last Blockbuster.
Dr. Lee: Oh, there’s still one left?
Convergence Training: Yep, still one left, and I read a little newspaper article about it, and I’m not sure to what they attribute their continued success. I’m sure it’s something–it’s not something like serving coffee or anything, but it’s something.
Side note: Here’s an article about the last brick-and-mortar Blockbuster, which is in Bend, Oregon (please note in the video I incorrectly said this was in Eugene instead of Bend–both are great places.)
I guess some other examples might be the hotel industry with Air B’n’B, taxis with Uber, Lyft, and such, and to your point, maybe a lot of industries might be facing that kind of thing, including learning and development.
So to that point, and I guess my next question, how do you see some of these disruptive technologies affecting learning and development professionals, either now or in the future?
How Are Disruptive Technologies Affecting Learning & Development?
Dr. Lee: I think technology is just one driving factor that’s affecting L&D. It’s coming to maturity now but we’re also experiencing globalization, we have a much more mobile workforce, we’re working with more people and there are many, many people coming into–I’m based in Canada, and I’m pretty sure it’s similar in the United States–there’s an influx of immigrants, and that also forces us to change the way we work. The aging work force is another driving force at play, and there’s a lot of talk about the Millennials, but at the same time the largest part of the workforce is 55+ and it’s getting bigger–that number if getting bigger in Canada but I believe in the United States it’s also substantial.
So all these implications are things we need to think about along with how technology is going to impact how we do work, particularly what role L&D needs to have. I think the fundamental question is not just how technology changes learning & development, but the more fundamental question is “What kind of problems are we trying to solve and how can technology support that?”
There’s a lot of talk in L&D about how we can get a seat at the table, and there’s a lot of talk about how can we remain relevant, so I think technology is part of that solution, it’s here to support that solution, but it’s not necessarily driving what learning and development needs to do.
I think we need to be smart about that. We need to stay on top of what technology can and cannot do, what are the pros and cons, and at the end of the day, technology may not be the solution that you need, but you need to be able to articulate that.
Convergence Training: Great. I like that your pointing out that technology is not necessarily a silver bullet that solves everything, AND that you’re saying that learning technologies don’t exist in a silo, there are all of these other things going on here too–a global workforce, an aging workforce, an increasingly mobile workforce, people working in multiple locations. And if it’s not exactly the same in the US as it is in Canada, with the largest percentage of the workforce being 55 and over, it’s certainly similar and a lot of our customers struggle with that as well (having a workforce that’s aging and nearing retirement).
So I like that you’re saying that we shouldn’t think of technology as being in a silo, and I also like that you’re saying that if learning and development wants a seat at the table and wants to continue providing value, we need to continue focusing on solving problems.
I didn’t plan on this question, but what kind of problems should we be focused on instead of simply being entranced by the shiny, glittery ball of technology?
Dr. Lee: Well, every organization, every business has their pain points, right?
If you listen, there are “fake” problems and there are real problems.
An example of a fake problem is when someone comes up and says “We need an email training course,” or “We need a training course on how people can run more effective meetings.” You need to dig a little deeper than that in order to understand if this is translating into a real pain point–is this something that matters in terms of your business goals and the strategic goals of your organization?
I think we need to get better at drawing out those questions and analyze it.
I believe, for example, when you’re talking about data-driven organizations or evidence-based L&D, that’s where that came from–is not just to take people’s “ask” when they say they need a course, not just to take that at face value. We’ve done that in L&D for so long. People come and say “We need a course on email etiquette” and we answer “OK, we’ll build a course for you.” I think that’s been going on for a very long time, and one of the best analogies I read elsewhere is, if you’re a patient coming to your doctor, and you say “I have this list of symptoms…my stomach is hurting, my back muscle is pulling, and so on,” the patient is not going to tell the doctor “And as a result, I need an operation.” No, right? You don’t provide a solution to your doctor. You tell your doctor your symptoms, and your doctor says”OK, based on these issues, I recommend these remedies or these solutions.”
The same should happen with L&D. We should not just take solutions people suggest and give the requested solution to them. Not every problem needs an operation, just as not every problem needs a course, and some of these “problems” are perhaps not even a problem, but perhaps it’s just something like people misunderstanding issues.
So I think, start with that mindset, it’s critical. Did that answer your question?
Convergence Training: Yep. I like that fact that you’re talking about digging down to real performance problems and not being someone who gets told to build a course.
The L&D professional Arun Pradhan makes the same point, he speaks at conferences on “How to Be Something Other Than an Order Taker,” which I think is what you meant.
Dr. Lee: Yeah, and I think in addition to that, we need to not just push back, but to push back with facts, with data, with evidence, with something to convince people. I think we need to be better at that, to be able to present both sides of the argument.
Thoughts about the Learning Management System (LMS)
Convergence Training: So, with the caveat that we need to be focusing on results and identifying and then solving real problems, and that technology isn’t always the best solution but it can be part of our problem-solving quiver, I was going to walk you some different technologies, both well-known and those that are just coming in. And the first I want to talk to you about is the learning management system, or LMS, kind of a classic. I wonder if you can tell us the state of the LMS and the LMS industry today and where it’s going.
Dr. Lee: Sure. OK, I think the LMS is still by-and-large the main technology that organizations use for delivering learning.
I know there’s a lot of talk about whether the LMS is on the way out and is it relevant, there’s a lot of debate about that, but at almost all organizations, that’s still the main platform for delivering training.
I think what I’m seeing more now is there are a lot more supplementary tools and technologies along with that. The LMS market itself, that’s sort of constantly in flux. I see a little bit more converging of the market space, this year particularly, the number of LMSs available dropped down a bit. It used to be 700 or so, and now it’s gone down to 600-something. Either they went out of business, or they pivoted and did something else, or they merged/got bought by different companies. My mentor and I keep a master listing of the LMSs that are out there, so we’re constantly updating and changing that, so we’ve noticed the numbers have gone down this year.
So, you know, and there is a lot more fragmentation of the market, so it becomes a lot more specialized, too. There are gamification platforms, LMSs that specialize in microlearning, LMSs that specialize in collaboration and social learning, LMSs that specialize in safety training, there are also industry-specific LMSs out there, right?
And at the same time, I also see there’s a bit of an identify crisis out there. There are LMSs that are really a CMS, a content management system, but it provides a lot of learning management function.
There’s also, this year, or maybe starting the beginning of last year, a new term–learning experience platform (LEP)–so I see a lot of these companies coming up from that perspective, and also content curation is getting quite big in that space. As far as AI and machine learning, of course they’re the buzzwords of the year in learning and development and particularly with LMSs, I have seen a lot more LMSs this year saying things like “We’re AI-enabled” or “We’re powered by machine learning recommendations for your learning path.” So these are some of the things I’ve seen this year.
Convergence Training: Great. I’m glad you mentioned artificial intelligence (AI) and machine learning, and that’s going to be something I follow up on with questions later.
Dr. Lee: Sure. I know, I don’t think we can have a conversation about disruptive technologies without mentioning that.
Convergence Training: Right. But I have of course heard of learning experience platforms (LEPs), but for those who haven’t, or even for those who are new to the idea of an LMS, I was going to ask if you could tell us what a learning experience platform, or LEP is, and how that’s different from an LMS, and maybe in the same follow-up tell us a bit about a learning record store (LRS) as well.
Dr. Lee: Yeah. There’s a lot out there.
The learning experience platform, or LEP, that’s still a little bit muddy. I don’t blame outside this field not understanding. I think those of us inside the field don’t understand it too well.
I think the way I see the primary difference is that the learning experience platform really focuses on the content curation part of it, in terms of how do you personalize and make recommendations for various types of content instead of just a system for administering and and hosting learning, there’s that focus. So that’s the main difference. And then of course there are additional little nuggets of features.
And a learning record store (LRS), a very new and elusive term, it came from–this is going to get kind of technical–it came from the back in the day when elearning didn’t have a standard. So we were trying to develop a standard and one of the main standards we began to use is SCORM, 1.2 and 2004, there are two SCORMS that are really similar, and it’s a still a standard that we hold near and dear to our hearts. The whole idea of having SCORM is just making sure that a learning activity or learning package would behave the same no matter what learning management system (LMS) you put it into. But the limitation of SCORM is that it only measures what is going on within the learning activity and the learning platform (or LMS) that you put it into. It’s a little bit limited in the way that you measure things. As opposed to the newer standard, which is known as Experience API or xAPI or Tin Can (though I think Tin Can as a term is going away a bit, I don’t see it as much as xAPI), so I think with xAPI the promise behind it is it’s much more integrated in the way you can measure things. So you can measure learning activities that happen outside an LMS, such as if you watched a video, you watched it for how long, you clicked on it, you paused where, you completed it, and what did you do next after that video? So, a lot of the trajectory learning, if you will, and also the various activities happen in and out of an LMS, so the learning record store is really a place to communicate and store this information.
Convergence Training: Is it your experience that most of these are built-into an LMS, bolted onto an LMS, or kind of separate?
Dr. Lee: Yes.
Convergence Training: All of that–alright.
OK, so great. Nice intro the LMS, and I guess another standard, time-tested, or older form of learning technology is the elearning course. I wonder if you could talk to us a little bit about that–its use and maybe its future as well. Can you touch on that a little for us?
Thoughts on the eLearning Course
Dr. Lee: Yeah, I think it’s still the thing to do, you know? The course design and the tools have gotten a lot better over the years. There’s a lot more choices in terms of tools you can use.
But I think by-and-large a lot of courses are still page-turners. It hasn’t gotten to the point where there’s a lot of innovation in courses.
What I see more is, in terms of courses, people are focused more on learning experiences, or more immersive learning, or more about learning moments, or chunks, or nuggets, right? So I think there is a move, shifting from talking courses, per se. And that’s true in the academic world, too, companies like Coursera and edX also are trying to shift their mindset too, in saying we don’t have to think of a course as being equal to “X” minutes of time, or it has to have “X” types of information in it. I think now we start thinking more a little about looking at the needs of learners, how do we support performance, looking at providing support at the moment of need. It’s not just about a course format.
You know, when you want to learn how to change tires, do you take a course? Or do you just need to know steps 1-5? Or do you need a mentor to talk you through it? So think about it from a performance perspective, what is something you can push? I think that’s why mobile learning, it’s been around, but I think now we also have the maturity of platform and availability of data and wifi access and all that to support it, and that’s why it’s popular, and that’s why microlearning is popular. That’s why sometimes game-based learning is popular, because it’s the interactive nature of supporting what people need to do.
So I see that it’s moving a little bit more in that direction, in terms of elearning courses. I don’t even like to talk about “courses” per se, to get people to think outside of that, to think about “Really, what do you want your target learners to know, and how do you know they know that, what kind of support can you give them and what kind of assessment can you test that they actually get it?” So that’s what I try to focus on.
Convergence Training: So that goes back to your original points about performance support, being aligned with business goals, identifying problems and solving problems, etc.
I like that you pointed out that there are a lot of tools that make this better, and that’s true, and we can do a lot of things, but also (and we see this at our work too) the increasing emphasis on performance support, shorter courses, microlearning like you talked about, delivering support at the time and place of need using mobile devices.
And I love your analogy about fixing the car tire. I have spent much of this year working with plumbing problems, just sinks that were dripping, and of course I’ve spent a lot of time on YouTube looking for the perfect three-minute video on how to replace that thing on my sink.
Dr. Lee: I’ve done that too, actually, and YouTube is my go-to.
Convergence Training: Yeah. For everyone at home.
OK, great. So…
Dr. Lee: Oh, can I add another point, though? Not so much courses, but the creation of courses and content, I also see more of a trend of L&D outsourcing instead of using in-house capacity. I think it’s just getting to the point where you can’t keep up, right? You can develop in-house capacity to build more traditional course-based learning, but I see a lot more specialization being outsourced to third-party vendors, and even just the volume of courses being needed, sometimes an in-house L&D department can’t keep up, so I think the trend to outsourcing continues.
Convergence Training: That’s interesting that you added your experience there. We see the same outsourcing to third parties at our workplace as well, and it’s easy enough to understand when you think of all the things an L&D professional has to do, or even we do a lot of work with safety trainers, and it’s the same thing for them.
OK, great, thanks for adding that.
So we talked about the LMS and about elearning courses, and in doing so we already mentioned briefly artificial intelligence (AI) and machine learning. I wonder if you could open that bag now and introduce us to some of the issues–talking about artificial intelligence, also known as AI, machine learning, perhaps explaining how those are the same and different, and then in particular if you could talk about:
- Content curation, which you touched on
- Adaptive and personalized learning, which you touched on
Artificial Intelligence and Machine Learning
Dr. Lee: Yep. So, artificial intelligence has been around for about 60 years, so it’s not really new. I think what’s interesting is now we have gotten to the point where technology can actually support that. We have cloud computing, we have the ability to host these large amounts of data, we have tools sophisticated enough to do analysis, so I think that’s why there are a lot of innovations in this space right now. And also applications that have started really tackling some problems.
Machine learning is a subset of AI, because AI essentially is how can computers fabricate intelligence, right? Intelligence such as the ability to make decisions or judgments based on observations or based on the availability of information to either select a path or analyze the content to make a recommendation or…there are all kinds of abilities for intelligence, as you know, but machine learning is a very specific way of explaining intelligence, of making decisions. It’s based on pattern matching, it’s based on algorithmic prediction to then come to a conclusion about things. So machine learning is more narrow, if you will, than what artificial intelligence can do.
And of course, even with artificial intelligence, you get more the general AI and the more specific AI. So if you think about intelligence as a general concept, it could be about intuition, about wisdom, it could be emotional intelligence, it could be all kinds of things. It could be the ability to draw inferences based on certain data points. And then specific AI is about understanding very specific sub-sets of that. And machine learning is more narrow, drilling back to basically using algorithms to identify issues.
So I think now we are more focused on machine learning, as opposed to AI, because AI is such a big field. So a lot of times, when people in L&D talk about artificial intelligence, they’re really talking about machine learning.
Definitely, you and I both know that in the past few years, discussion of that is all over the L&D space, in terms of how we can use it, what are the implications, is this going to replace jobs, how do we regulate things?
One of the big topics right now is about ethics or accountability. Not just at the company level but at the government level, international regulatory body level. We know that companies like Google, SAP, Microsoft have now what they call AI principles for ethical use, but they’re companies, so they still are primarily putting these out based on the best interests of the company. What I’m interested in seeing is at the government level, what kind of thoughts do we put into that.
And why does it matter? Because you when talk about AI and machine learning in L&D, what do we use it for? A lot of the times, we use it for personalized learning, right? Understanding how a learner or an employee is performing, perhaps, in the LMS, and based on this person’s ability to complete or not complete a certain learning, for example, that might provide an intervention or provide support or do some coaching for them. So the danger for that is (a) when you’re collecting this type of data, who owns it? Are your employees aware that they’re being identified and analyzed at that level of detail? And (b), the implications for that person–is that information linked to their performance review, for example? And of course, the other danger in that is are you stereotyping this person? Perhaps this person is having an off-day or an off-week; sometimes past behavior is not the best prediction. You know, we’re human, we go through ups and downs. I know we’re wary of stereotyping people, but there’s a danger in that, too.
And we haven’t really talked about that, as organizations we haven’t really looked at it, as countries or as international governing bodies…I understand that the UN is developing some sort of guidelines on that, but it’s not out there.
Another danger is when you stereotype, and you say “this person is at risk” (there is an LMS out there that puts people into risk quadrants, for example), whether you’re failing or succeeding, right, and it’s a prediction to say something like “based on the number of logins into the LMS you’re at risk.” But just because this person isn’t logging into the LMS frequently doesn’t mean they’re not learning–perhaps they don’t need as much time, or maybe they downloaded the material offline, and maybe you don’t know they spent all that time. So those are really nuanced things that are hard to know, and I don’t think that machine learning and AI are “there” in being able to measure those things, and we haven’t come up with an overarching plan to say “This is how we measure that” and “If there are gaps, how are we going to mitigate those gaps?”
So those are my concerns, and those are the challenges I see.
Convergence Training: So, I do want to get back to some opportunities, but I love that you talked about the challenges as well. And one challenge I’m aware of, only because I’ve heard you discuss it in the past and I’m hoping you’ll discuss it again today, involves a “woman” named Jill at George Tech…
Dr. Lee: Jill Watson, yes!
Convergence Training: …and this has to do with the use of machine learning and what the machine learning can do based on the information coming in.
Dr. Lee: That’s right, but before I talk about that, I want to make sure I also present the promises with AI, because it does have tremendous opportunity for us to personalize or support learning in a way that maybe we weren’t doing in the past.
Not that we as instructors weren’t providing learning in a personal way. We do this all the time, right? If you look at a classroom of learners, and if it looks like they’re falling asleep, you might change your tone a little bit, right? That’s personalized learning.
But I think it can help in a large scale, because you can’t personalize for a 500-person audience, but a machine can. It can push out learning based on certain characteristics of your learning. So to scale, it’s helpful to look at analysis of the learning data to help interventions, supporting course designers is always great. As long as the power remains with humans to change that, and not having a machine overriding and making assumptions about things.
So, about Georgia Tech, this is an interesting story and I love it. For those who don’t know, a few years ago, for one of their online Masters level computer science courses (the course is huge, it always has more than 300 students enroll), so they always have teaching assistants to help answer questions on discussion forums–the students post questions, and most of these questions are pretty mundane, like “When is this project due” or “How many pages do I need to write” or “What citation format do I have to use?”
So Georgia Tech traditionally hires a lot of TAs to help answer these questions from students. And in 2013 or 2014, they decided to introduce chatbot to their course to help. They were still going to have human TAs, but they added a chatbot TA without telling the students. And they decided to call this chatbot TA “Jill Watson” after Watson, the IBM computer.
So Jill Watson was there along with all the other TAs for the duration of the semester, and she did such a good job that students (a) didn’t know she wasn’t human and (b) students were like “She’s great! We should nominate her for best TA of the year.” So it was quite interesting.
But the challenge of using that is you can only train and feed information into a robot based on the information you have. And so you can only feed information to this robot based on the questions students have asked in the past. So what do most students ask? Things like “When is this assignment due” or “How many pages do I write,” those kinds of things, and along with questions like that, you want to train your robot to handle conversations that are more personal, that involve more conversational data, and this is where it falls short.
Most students in the past were male students, so there’s a really interesting paper out there called “Jill Watson Doesn’t Care That You’re Pregnant,” and it talks about when a male student talks to Jill Watson, and says something like “I just found out I’m going to be a father soon,” Jill Watson will reply and say something like “Congratulations for welcoming your bundle of joy!” She had a very appropriate response to a male student talking about the experience of being a father for the first time. So, when a female student said a very similar thing to Jill Watson, saying “I just found out I’m pregnant, I’ll be due before the end of the semester,” Jill Watson had no data set from the past to inform her how to respond appropriately so she just said “Welcome to the course!”
Convergence Training: That’s a great story–thanks.
That’s a great story for two reasons. One, you have to get the appropriate information in for your machine learning, which is maybe being delivered by a chatbot so it can give an appropriate answer, but second, obviously, a bigger societal issue that the people who develop these things may have their own biases, including gender biases, that you’re cautioning against.
Dr. Lee: That’s right, yep.
And sometimes they don’t know their biases, right? I mean, the reality of this data set is that there weren’t that many female students to use to put that information into the system.
Convergence Training: Right. I think we’re all unaware of our biases, largely.
OK, so. We talked about machine learning and AI. We talked about the fact that it has some challenges, and I loved your Jill Watson story. But it also presents some opportunities, and I think we’ve pretty clearly pointed out at least some opportunities for personalized and adaptive learning. We’ve indirectly mentioned the chatbot opportunity, that’s machine learning powering Jill Watson and other chatbots. Are a lot of companies using chatbots for L&D to support employees as well?
Dr. Lee: Yeah, yeah, lots. Slack, a communication tool, has it. There are a lot of chatbots, especially in the HR space. I think there’s a chatbot called Donut that organizes your coffee meetings for you. There are also some in the form of virtual courses and mentors, There are chatbots that do onboarding, that push out information from Google Drive and Evernote and Dropbox, you know company information is often in silos, so the chatbots can source information from all these different back-end resources you feed into it to then prioritize and push out the information to your end users.
Side note: Here’s an interesting/helpful article on integrating machine learning into Slack.
Convergence Training: That’s great. I can see how that would be useful. I was just discussing how I get confused between Google Drive and everything else…
Dr. Lee: Yeah, right. Because especially when you’re new to a company, or even if you’re an experienced employee, you don’t care where the information is stored, you just want answers when you need them.
Convergence Training: Right. I only care because I can’t find it because I don’t know where it is.
Dr. Lee: Exactly. And you just want to ask that question, right? You just want to say “Where’s that safety training information from 2015 that somebody created?” You just want to put that question out there and have something look at every thing for you.
Convergence Training: Yeah. And then, maybe that’s related to this next issue. You also mentioned machine learning in the form of content curation. I wonder if you could explain, for people who don’t know that term, what “content curation” is and how machine learning can help facilitate that?
Machine Learning & Learning Content Curation
Dr. Lee: Oh yeah, sure.
So content curation, as opposed to creating content from scratch, you can think of curation as being like an art museum. In an art museum, you don’t paint your own paintings, you go and source paintings from around the world based on a specific theme or a specific focus, right? You want the Impressionists, you can be very specific, saying you want The New York School/Impressionists/of Jewish descent/painters, right?
So you can be narrowly focused or you can be broad, but the point is that you look at it from different sources out in the world, and based on your theme or what your needs are, you take it from different places and put it into one place, and you offer that to your audience.
Convergence Training: And I assume a big part of that has to do with tagging and searchability and findability, and we all know that companies like Netflix or Amazon do that and L&D professionals need to do that better, partly so learners can find that information they need in order to get that just-in-time performance support we’ve been talking about. Is that correct?
Dr. Lee: Yeah, I think a larger part of it, though, and this is where the promise of machine learning and AI come into play. We don’t need AI or machine learning to curate content for us, necessarily. As humans, we can manually curate, and we’ve done that for hundreds of years.
What machine learning is good for, based on my curation or me as a learner, based on how well I am interacting, how I am responding to the content that’s being curated for me…First of all, content can be curated just for me, me as a unique individual.
Convergence Training: So personal curation.
Dr. Lee: Yes, personal.
And secondly, machine learning can analyze based on my interaction with curated content, is this the right content? Because we don’t know…sometimes you may be curating content that’s WAY above my knowledge level, and it’s very difficult to read, so I give up after two minutes.
So what machine learning can do is analyze my behavior with this curated content, say”Hang on a minute, Stella is not enjoying this, she’s looked at it for two minutes and she never looked at it again, she tried answering a few questions and she’s not doing a good job answering these questions, so maybe we need to go back and adjust the knowledge level to give her something a little bit easier or a little bit differently than what we’ve been giving her. So that’s what machine learning can do.
Now, what I think is better, and I see this happening now, is for me to be in control of that as an end user. To say, “I don’t really like this, this isn’t really helping me, I need to learn to change tires, but I don’t need to know how to change bus tires, stop giving me videos of bus tires, I just want to learn to change tires on my car, so stop giving me all these other videos that aren’t relevant.” So I can push back and say “Yes” to these and “No” to that and perhaps these other things, so it puts control back into the hands of the learner, which is very powerful, and I do hope that companies will do more of that. Which, obviously, Netflix is not very good at doing.
Convergence Training: That’s a great explanation, thanks.
I was focusing on that use of machine learning for curation, but I like that you mentioned you can use it to select the right content but also the right level, and I like how you said “Hey, let’s not get lost, having robots running the world, and maybe it would be nice if the learner could tell it what kind of content the learner likes.
Dr. Lee: Well, I think my take is always that machines are here to collaborate with us. So we need to have that two-way collaboration. We humans are good at certain things; machine are good at certain things. We can make it much more powerful, we can focus on what we each are good at, with that interaction that goes back and forth, right?
Convergence Training: Yeah. That’s fantastic, I love that point.
And that’s try in L&D but we also see that at work with manufacturing workers, asking “How are you going to have a job alongside that robot, you know, that pallet stacker?”
Dr. Lee: You know, one of the manufacturing tools I’ve seen is the ability to augment your reality. Right? Like the ability to overlay something on top of your reality with an iPad, to say “For this piece of equipment, I can get all this information by overlaying my iPad and they push all this information to me,” I see that as a tremendous application, being able to overlay information on top of machinery.
Convergence Training: Yeah, year.
So you expertly and adroitly anticipated my next question.
So if there are industry buzzwords about machine learning, and artificial intelligence and xAPI, there are also buzzwords about virtual reality (VR) and augmented reality (AR).
I wonder if you could quickly describe each of those terms and then explain how they’re being used and what are their potential uses?
Virtual Reality (VR) and Augmented Reality (AR)
Dr. Lee: Yeah, so the way I like to explain the difference between virtual reality (VR) and augmented reality (AR) is that you can think of virtual reality as immersive. You leave your current world behind and you go into another world, completely. So, you’re immersed in, or it will transport you to–Italy (or someplace else). That’s not this world, you go to Italy. Or it immerses you in a soccer field, you’re not in a high-rise apartment, like I am.
Convergence Training: Or, I’ve seen a great one where you’re immersed in a simulated underground mine escape. You’re in a mine, underground, with your coworkers, you can interact with them, and you need to get out.
Dr. Lee: Yes. Or a fantasy world. You can be in some sort of kingdom somewhere. Like you can be in Lord of the Rings. And so you can be completely somewhere else.
Augmented reality (AR) you’re still in this world, it’s still here with you, you still see your surroundings, but it’s overlaying something on top.
So an example would be what car manufacturing companies are doing with their windshields. I don’t think it’s to market yet, but BMW and other high-end vehicles are experimenting with overlaying road conditions on the windshield as you are driving. So that’s an augmented reality application–you’re adding a layer or layers on top of your current world as opposed to leaving your current world and immersing yourself in a different world.
Is that clear, did it make sense?
Convergence Training: Totally. And I think the car windshield thing is a great description, and we probably saw that 30 years ago in Star Trek.
So in that second thing, where it’s augmented reality. That additional layer of information that is augmenting your true external reality–that’s usually being seen through some kind of mobile device, like a tablet or a phone, is that correct?
Dr. Lee: That’s right. Or even there used to be an augmented reality company here, I think it’s been sold, but they design ski goggles and sun glasses. You know, I live in Vancouver, and Vancouver is very outdoorsy, so there’s a huge market for these kind of things. They provide additional information for you–the sunglasses are mostly for cyclists, so they’re wrap-around sunglasses and they provide you light updates and traffic conditions and stuff like that–outside temperature, and all that stuff. So that’s a huge application with that sort of thing.
Convergence Training: And I guess in a similar way, if that’s a great addition for a bicyclist, I’ve seen similar uses with what we used to call Google glasses and we’re now supposed to call smart glasses, I think…
Dr. Lee: That’s right!
Convergence Training:…but I can put these on, as a machine technician, and I can look at the machine in front of me, but I can also see a procedural list on the glasses themselves with a procedure I follow to operate or fix the machine, and I’ve seen some studies that show that can really increase the speed and accuracy with which these job tasks are performed.
Dr. Lee: Jeff, you’re actually recording this conversation from your glasses, right?
Convergence Training: (Laughs) Yeah. 🙂
Dr. Lee: You know, there actually are augmented reality glasses where you can push record.
Convergence Training: That would be super-helpful. And I think I’ve even seen cases where the glasses are delivering live video so it can assist the worker in the field as well.
Dr. Lee: It’s quite distracting, though. I tried it a few years ago and I nearly walked into a wall just trying to control it. So you sort of move, and you’re making an adjustment, and you’re not familiar with the physical world around you. So there’s a danger with that as well.
Convergence Training: Well, I am glad you pointed that out. So, opportunities and challenges. So, with these VR glasses, am I correct that there’s also an issue with people getting nauseous or dizzy?
Dr. Lee: Yeah. It is largely device dependent. As a general rule, the cheaper device you buy, the bigger problem with getting nauseous and dizzy.
If you see a gap between your glasses and your face, that’s a problem–because you want to be immersed in that reality.
I think it’s also a problem because you can’t be in that world for too long. It’s not something you can sit around in all day. It’s disorienting after a while, and you need to come out of it. So for learning, you need it to be very targeted. You don’t want your learner to be sitting there learning something for four hours. You just want to target it to, maybe, a difficult concept, something that’s much easier to explain it when the learner is immersed in that reality.
I’ve seen a lot of good applications (of VR) on empathy training. That’s something I find tremendously helpful, because it literally puts you into the other person’s shoes, you’re not just hypothetically thinking about it.
I love that, I think it’s Stanford, they have a VR program basically about understanding the homeless (see Becoming Homeless: A Human Experience) and it’s very impactful. It tells a story, right? You’re in your apartment, and you lose your job, so you have to sell your furniture, and you’re literally seeing your furniture going away piece by piece, and that visual stays with you.
Convergence Training: That’s great.
I hadn’t really thought much about that use of virtual reality, and of course empathy and other soft skills are important 21st century skills.
I guess on the flip-side, in manufacturing or in safety, you can use that VR-enabled training for situations that might be too dangerous…
Dr. Lee:…that’s right, or it might be too expensive to shut down the equipment.
Convergence Training: Right.
So, to make a connection with something we mentioned earlier and to set us up for the next point, I’m assuming, especially with virtual reality, where I’m in a fantasy world and I’m interacting in different ways, I can use something like the Experience API (xAPI) to catch a whole bunch more data points generated during training, and then have a whole lot more data to analyze Jeff’s performance in that virtual world. Is that correct?
Dr. Lee: Yeah, hypothetically speaking, you can be capturing that, and you should be able to transfer that into some form of referent, right?
Yeah, that’s a good point. I never really thought about the implications for measurement, to be honest, in a VR world.
But yeah, especially there’s a have-to response, if you have to do something with your hands or with the VR objects, like the interaction with something like heavy equipment, what are the points of interaction.
Convergence Training: Right, right.
That’s something I find fascinating about VR in training. Obviously training is all about teaching people to make good decisions, to have good behaviors, to perform well, and there are all sorts of potential data points in one of these virtual reality courses that we could be using to extract training data..
Dr. Lee: One interesting application I’ve seen for training soft skills is there’s a platform you can use to learn all sorts of business sort skills, like networking, making presentations, and public speaking. And of course, this is a pay service, there’s some free stuff but it’s mostly pay, and you put this one platform where you can record your speech–let’s say you need to make a presentation, you can practice, and they will record it for you, and they’ll analyze it, based on say the number of pauses, or based on your pitch, based on your enunciation, so it gives you some sort of analysis back.
So I imagine you can do that with any other VR training programs, so if you give it a recording or series of recordings, it will give you some sort of analysis back.
For more, see this Virtual Reality and Augmented Reality Explained article.
Big Data and Data Analysis
Convergence Training: OK, so perfect. So that leads us to BIG DATA and data analysis, yet another buzzword of the year.
Maybe you can tell us a little about that?
Dr. Lee: (Laughs) I feel like you’re going to cover every single topic of the year!
So, I think big data, in the learning world, is mostly in the form of learning analytics (sometimes you see LA for this). It came primarily from the world of academia. There’s this huge research in that. But I see a trend slowly creeping into the corporate world now. There’s this huge effort saying “We have this body of research and now we need to apply it.” So there’s a bigger connection in that space right now.
A lot of the LMSs I think are doing a better job at leveraging big data and leveraging learning analytics. I think the interesting point is that people are still not really sure what to do with it. People are still trying to understand it. It’s almost now like the technology gives you all these options, and it’s overwhelming, and the bigger piece now is “How can we actually teach people to understand that?”
Of course, I’m always skeptical about big data: are you sure you want that? The issues I see are that people end up just measuring things that are easy to get, just because they’re available, it’s the easiest thing. So they’re like “Oh, uhm, this learning is effective and impactful because it has a 98% completion rate” or “there’s a large enrollment,” because those are numbers that are easy to get and it’s a no-brainer. But is that meaningful to you? Perhaps not.
And I think that’s where we need to be deliberate about measurements. We need to be deliberate about asking “Are these the questions you want answers for?” So I think we need to get better at knowing what questions we need answers for. So if you want to determine what learning is impactful, we have to know what that means. Impactful–we need to define it, we need to figure out what are these questions so that big data can help answer these questions.
And sometimes, just because we can measure everything doesn’t mean we SHOULD measure them. I think there’s also a challenge about people who just wanted to do something with big data. You know, it’s like when you have a hammer, everything’s a nail. So everything is based on data analytics, and here are these beautiful diagrams of everything we don’t need.
So I see a lot of challenge in that, but I also see there’s opportunity to align big data with more qualitative information. So sometimes I like to call it “small data” to sort of cut it. You know, with big data you get the breadth, and so you want to go deeper and perhaps talking to your users, having a few focus group studies or interviews, that would complement the big data really well. So I think we need to think of the combination of your qualitative and quantitative measures.
And that’s where big data will be useful, to sort of give you that first cut: “Here are some patterns we detected. Are these true, are these assumptions valid?” And that’s where you have to go and validate them by perhaps adding on some qualitative measures out there.
Just because there are patterns doesn’t mean they’re true, right? We do this all the time…we make assumptions about things. Just because this one tree is turning yellow doesn’t mean it’s autumn-maybe that tree is dying. So I think there’s a danger in big data is thinking “These are the patterns so it must be true” without critically analyzing it. But in general, I see in L&D, that’s the thing we’re least good at. But everywhere else in the business world, that’s one of the top driving forces, to be a data-driven organization, but in L&D that skill set is lagging, and something we need to develop, and the skills are getting easier and easier, whether that’s inside or outside an LMS,
You don’t need to be a data scientist, I also feel that as an L&D professional you shouldn’t feel insecure about that, you don’t have to go and take data-science courses, we don’t need to be scientists. But I think we need to have a fundamental understanding, we need to partner with IT, we need to know what are the questions to ask, and we need to apply that to what we do.
So, that’s my take.
Side note: Check out Dr. Stella Lee’s article The Case for Small Data in Learning here.
Convergence Training: OK, so I like your distinction about small and big data. Stella has written a nice article on that (see link above). And I like your point about how we’re in a world where it’s easy to get data, in fact we’re flooded with it, and some of that data is easier to get and some of it is harder to get, and we can get entranced by the data that’s easy to get, like course completion rate or how long were people in courses, and maybe that’s not impactful, and we’re getting now into the title of that Nate Silver book The Signal & The Noise, and we can not be looking at the signal. And to your point, we should be asking which of these data are really showing me that my learning is impactful, and you had a lot of cautions about what’s noise. Any tips about what kind of data people should be looking for that’s a signal, that shows that learning is in fact impactful?
Dr. Lee: I think that’s very contextually dependent though, right, depending on what kind of learning you’re doing and what you want your learners to know.
I guess my advice based on experience is “Yeah, you can measure course completion and all that stuff,” but really, what do you want to know? How do you know your learners learn, right? So if you’re learning to, say, change tires, the fact that somebody watched a YouTube video on changing tires doesn’t mean that person can apply that.
So that’s the first level, right? The requirement is you watch this video. But the second requirement is you need to observe that person changing tires. So you need to actually go and collect that data onsite when that person is doing their job, right? And that’s something that technology can help with, because you can say “Put sensors in the person’s gloves while they’re changing tires,” right? You can quantify anything, if you want to.
However, I always caution people about the opportunity cost. Is it worth spending lots of money to design something to put sensors in a pair of gloves so you can measure how people change tires, or would it actually be more cost-effective to just send somebody to observe it, right? So you can get there many ways. But you have to think about the opportunity cost.
The same is true when we’re talking about virtual reality (VR) and augmented reality (AR) development. They are very expensive. Any of these learning programs…any learning activity you put in a VR world is at least 100K in development costs….I’m not even talking about implementation or buying hardware or anything else. That’s a chunk of money for one learning activity.
And the same with measurements. It costs money to put measurements on certain things, like if you’re going to put sensors into gloves to measure how people are interacting with machines. And perhaps that expense makes sense, because maybe you don’t have enough qualified people to observe…maybe you’ve only got 1 qualified person to handle observations but you can easily produce five pairs of these sensor-enabled gloves to re-use over and over. Yeah, that’s a good business case, it makes sense, it’s appropriate for the kinds of things you want to measure because you can argue that the sensor picks up exactly the same type of information that a person would observing others changing tires.
So in that case, that’s appropriate and the right thing to measure. But that doesn’t mean I can measure soft skills in the same way. So this measurement issue depends on the type of learning, the type of outcome you want, and we have to think about that accordingly.
Convergence Training: Good. That’s great to hear you connecting where we were in the beginning, trying to connect learning with business goals and job performance, so I love that.
Well, Stella, it might have seen I was going to drag you through everything I knew about learning technologies, and you’ve been super generous with your time and knowledge.
Side note: I should have asked Dr. Lee about streaming video and block chain but forgot 🙂
I’ll see if I can ask her about those on LinkedIn…
I wonder if in closing, is there anything I should have asked but didn’t, or do you have any final words on the use of technology in learning or any cautionary tales about over-reliance on technology or anything you want to close with?
Closing Thoughts on Technology and Learning
Dr. Lee: I clearly love this field. I’m skeptical, as you mentioned, about the “silver bullet” and about jumping into technology without really understanding it. But I’m by and large very optimistic, and I think we live in a very good time. It’s a privilege, you know. You have so many choices. It’s a privilege to be able to help people with all these tools and supports and all that we have. I think we should all be excited about it.
And also at the same time, I’d say don’t be afraid to experiment. Don’t be afraid to try things out. Don’t be afraid to work with your stakeholders to see if they’re open to trying something different or if they’re willing to test something out. I think we should have that mindset of having an experimental approach. I don’t mean just piloting a course, right? Just say “Let’s take this in a different direction, let’s look at this in a different way, let’s understand how we can solve this problem, is this even a learning problem?”
I think just being able to open your mind, to think about these possibilities. I think it’s tremendously exciting. I also think we are more relevant and needed as learning professionals that ever because of all these conversations about the future of work and about the changing technologies and globalization. Do you know what these things always get back to? We need more training, we need more learning, we need more reskilling, we need more upskilling. These are all L&D functions. it’s always getting back to us, so I think this is an opportunity for us to take on a bigger role, to step up and really be part of the organization and to drive that change.
Convergence Training: Those are great points, thanks.
For everyone out there, I want to say again that this was Dr. Stella Lee, and that Stella is with Paradox Learning. Stella is really active on LinkedIn, it’s easy to find her. Stella, are you active on Twitter and Facebook as well?
Dr. Lee: I am trying not to use Facebook, but on Twitter I am StellaL. I find it much easier to engage with my audience on LinkedIn, so look for me there
Side note: Here’s Stella’s LinkedIn profile.
Convergence Training: Great. Hope you all learned a lot from Dr. Lee and we’ll see you in another podcast soon. Have a good day.
Dr. Lee: Thanks Jeff, thanks.
Conclusion: How Technologies May Affect Workforce Learning & Development
Wow, what a great interview with Dr. Lee about a lot of the technologies currently involved in workforce learning and development (or coming soon).
A million or perhaps a jillion thanks to Dr. Lee for her time and expertise–so appreciated!
If you’re a reader or a listener, and you’re hoping to integrate some of what you learned about using technology in L&D, we wish you good luck and invite you to share your, thoughts, experiences, and lessons learned in the comments below. And remember, as Dr. Lee said, don’t be afraid to try things out.
And before yo ugo, don’t forget to download the free LMS Buyer’s Guide below.
Learning Management System (LMS) Buyer’s Guide
Learn what you need to know BEFORE you begin your search and get a free checklist to guide you, too.