Machine Learning in Corporate E-learning - Applications and Trends

Editor's Notes

• #2: Good afternoon. Thank you for having me here. I hope you had a good lunch.
• #3: So a little bit about my background. My entire career focus has been in e-learning. I have been going at it for nearly 20 years now. Starting in academic institutions, helping universities to develop large online curriculum and integrating technology into teaching and learning. Then I moved into private sector and government. The last four years, I have been running my own consulting business called Paradox Learning and that is what brought me to Vancouver for the last two years. I lived in five countries and consulted with many more – so I am always interested in hearing about how different technology is being used or how technology is being used differently from around the world.
• #4: To set expectations for today’s talk, I am going to give a high level overview on how machine learning and AI are being used in the corporate e-learning world. With that, I will go over some background on why this is happening, as well as showing some examples on current trends and applications. Just to be clear, this is not a “how to” technical talk, more of a survey what various applications are out there and some of the limitations. Hopefully, it will inspire you to design some yourself.
• #5: So you might want to know why you should care about this topic? In general, there has been a lot of hypes about machine learning and AI in general, if you read anything at all, you can’t get away from this topic. Specifically, one topic of concern is about the future of work. A recent PwC study says that 38% of the jobs in the US are at high risk of being replaced by AI over the next fifteen years. It is hard to pinpoint the exact number of jobs being impacted and that number seem to change depending on which technology expert you talk to.  Even though these numbers are arbitrary and meaningless, we ought to think about the larger question – the workplace is changing rapidly. More than ever, organizations are desperately trying to understand how to engage employees better, how to develop their talent, how to transfer knowledge as people retired, provide intervention and performance support when needed or even before they are needed, and in general to prepare them for the future of work.
• #7: A quick scan on “the future of work” on LinkedIn returns nearly 40,000 results. This has major implications which require investments in training and reskilling of employees by the organizations.
• #8: For the past decade or so leading up to now, organizations have invested in learning technology primarily in the form of Learning Management Systems, or simply known as LMS.
• #9: This is a pretty standard interface for LMS – you put your courses and related learning material on there, sometimes it will have some social features such as discussion forum, and usually a way to display learning data by way of a dashboard for learners, instructors, and administers to gain insights from.
• #10: This is an example of a course shell for self directed e-learning. The majority of corporate learning is designed in this format whereby all learners go through the same material in a linear format – more or less.
• #11: Here is an example of some learning data. Tracking is mostly focus on completion and assessment scores. Limited learner data. Passive data collection. Not asking users for input/users don’t have a lot of say in the matter. Basic descriptive data – averages, sum, counts, rankings, high and low points, percentage changes, etc. Intervention opportunity is lacking – no insights into how learners are doing when they are going through the learning material.
• #12: Another example of a learning dashboard
• #14: SCORM stands for “Sharable Content Object Reference Model”. Essentially, it is a set of technical standards for e-learning software products. It is for e-learning interoperability.   SCORM tracks course completion status and assessment scores but not much else. xAPI allows a granular look at various learning activities – whether a learner clicks on a particular link, how long did one pause at a video, etc. Those data feed into adaptive systems/recommenders. This provides a more targeted resources to your learners. Learning data can be further analyzed if integrated into HR systems. Past performance data (coaching, mentoring notes/checklist/rating, performance review notes – sentiment analysis) + LMS data (course-based data + some social data)
• #15: By the way, when I was looking up some references on AI and tin can api, I got this search result…
• #17: According to the 2018 Learning and Talent Platforms Buyer Study published by Elearning! Magazine, personalization closely followed by learner engagement are top business reasons for investing in learning technology for companies. Essentially, it can be broken down into four areas of applications.
• #18: Chatbots, as you know, are nothing new. In the old days, chatbots are confined to a pre-defined set of answers. But things are getting a lot more sophisticated now, and are being integrated as a larger part of corporate learning strategy, especially in the area of knowledge management. They can find answers in context – act as a quick reference guide. Use for coaching and performance support, like an interactive job aid, or a virtual coach, pushing out info and providing feedback. Single source of information and resources for internal team to access. Integration with Google Drive, Evernotes, Dropbox, and a few others for source of information Built for Slack Can push knowledge out – what they called Flow Canadian company from Ontario!
• #19: An example of an onboarding conversation to take a new team member step by step. It goes to various data source to push out information
• #20: Generate personalized learning paths, self-paced, micro-learning. Monitor learner progress and predict which learner is likely to want to learn next. Employee engagement data – machine learning provides a more effective way to analyze big data and to identify patterns that suggest content could be better written, completely redesigned, or to provide additional support to your learners if they are failing to complete a course or a learning activity.
• #21: Change content on the fly No-Touch Individualization feature Instantly evaluate the efficacy of learning materials, and maps existing content to assessments and regulations Learning Intelligence Proof of concept program - collects real-time, fine granular behavioral data of up to 100 learners without a full technical integration.
• #22: Pinpoint Pattern – e.g. significant spikes in course failing. Intervene before it is too late. Some of you might have heard of D2L? D2L has been rebranded as Brightspace. They target both the academic and corporate markets. Performance Plus is a package you can purchased and add on to the LMS. Some of the generic indicators within LMS for learner retention and completion rates are: Recency and frequency of login, participation in discussion forum, assignments turned in on time, even clickstreams In academic world, they also would factor in GPA, overall grades/curving of grades, SAT scores, etc.
• #23: Predict and present the most relevant content to learners by automatically creating personal learning paths. Identify at risk learners by predicting learners’ final grades for a course. Visual dashboard with Risk Quadrants. Give instructors more actionable insights.
• #24: The indicators are normally around frequency of log in, duration of log in, grades, engagement in discussion, etc. Do not measure quality of interaction
• #25: Provide personalized feedback to massive amount of open-ended assignments Detects the response patterns in essays and sort them into piles. Might work better in certain subjects over others.
• #26: Classification could be tricky Not sure if this is adding more work to the teachers as you need to double check the classification Wonder if you can tweak the algorithm
• #27: These are all fantastic, but by no means perfect. I want to share with you some of the challenges and limitations. I would also love to hear your ideas on this.
• #28: Prediction could be too prescriptive. Also they could demotivate learners, or be a barrier in accessing learning that your learners truly want. Learning behavior is very tricky to predict. Learning is very hard to recommend/personalized. Machine learning and big data analysis by no means can replaced instructor observations, and feedback gathered from learners, peers, and their managers. Adaptive learning is also very time consuming to build. Just because a learner preferred a certain type of learning (e.g. video-based learning), it doesn’t mean this person always prefer it. Largely context dependent. If I am at a noisy café, I might just want to read the document. If I am learning a foreign language, I might want to listen and practice. Past behavior is not always the best prediction for future learning preference. Adaptive learning is hard to evaluate – how do you know what you miss? Trust – companies don’t trust machine learning algorithm. Lack of governance, lack of transparency. Algorithm black box – would be helpful to have the machine to be explicit about the decisions it makes to recommend/not recommend certain learning paths or options over the other. Humans need to actively engage in the decision making process. Scale is an issue – not going to work for a 30 people course over six months. Data not generalizable.
• #29: I will leave you with this quote from Ginni Rometty, the current president and CEO of IBM.