Measuring the ROI of learning: Just a moonshot?
Measuring the ROI of learning: Just a moonshot?
Five ways to assess your return on investment.
How ambitious are you with your projects? And what return on investment do you get for them?
These questions were front of mind on September 12, 1962, when U.S. President John F Kennedy stood in front of 40,000 people in a stadium at Rice University and set fire to a giant pile of money. Not his money, of course. No, it was the money of the people watching him, and those watching at home: the people of the United States.
He did it with these words:
‘I believe that this nation should commit itself to achieving the goal, before this decade is out, of landing a man on the moon and returning him safely to the earth.’
The initial cost projection for the project was $22 billion (equivalent to $172 billion in 2023). The total that NASA spent between 1960 and 1973 ended up being far higher: $49.4 billion ($482 billion adjusted).
The grumbling was immediate. Former President Eisenhower called that level of spending ‘nuts’. In the Senate, there were concerns that the space program would take money away from military spending at a time when the Cold War was getting hotter.
In the following years, NASA’s budget grew to consume 4% of total U.S. Government spending. JFK, assassinated in 1963, never saw his goal achieved.
But the project was a success. On July 20, 1969, Neil Armstrong became the first man to walk on the moon. On July 24, he and his crew returned safely to Earth.
Mission accomplished.
What was the return on this investment?
We’ve been thinking a lot about ROI lately here at Mind Tools Towers. We’re entering awards season, so we’ve been reviewing client projects to identify worthy contenders. Our pitches and tenders often include some element of ROI. We’re making decisions about where best to invest in our products, and so that requires analysis and trade-offs.
And a recurring refrain from many is how difficult it is to measure the ROI of learning.
Upfront, we have no crystal ball. We cannot say with certainty what will happen. Afterwards, it is difficult to isolate the effect of a single learning intervention on business performance.
But there are some techniques we can use to make estimates.
💰 If the goal is financial
The easiest projects to measure are those with a financial outcome. If you are introducing sales training, you can track sales before and after your intervention. Better yet, you can provide the training to one group of salespeople and compare results with those who did not do the training. Or you can ask participants to estimate the impact of the training on their sales performance and use an average to make a ‘best guess’.
If results are heading in the wrong direction, that’s also a signal to take another look at the training itself.
⏱️If the goal is to save time
Another relatively easy approach is to look at cost savings. Our award-winning blended learning programme for Scottish Enterprise paid for itself in two years by replacing face-to-face workshops with a mix of self-paced e-learning and online sessions. It also had a financial return in so far as participants reported increased revenue for their businesses.
A few years ago, we worked on a major compliance programme that paid for itself within one year by halving the time that employees spent learning. In essence, we worked with individual subject matter experts to make the courses more focused. ROI was based on time saved per employee, multiplied by the average hourly salary. And, of course, we belive the learning outcome was at least the same - if not better for being more focused.
We’ve written before about how to reduce compliance costs.
📈 If there is historic data we can forecast with
We can also compare the impact of ‘doing something’ with ‘not doing something’. Suppose you are concerned about your organization’s employee retention rate. You could look at monthly recruitment and onboarding costs for the past two years and project those forward on the assumption that nothing changes.
Then, as you deploy your learning intervention to reduce resignations, you can look at how those costs change. The difference between actual costs and your forecast represents your ROI.
🛠️ If re-work can be reduced
L&D often get tasked with stepping in when processes are not being followed. What is the cost of this issue?
If you can gather data on the financial impact of re-work, delays, remediation actions and so on, then you can start to plot out the ROI of fixing those issues. You might also decide, depending on the answer, that it’s not worth fixing them.
🔍 If you’re launching a multi-purpose solution
The ideas above are ideally suited to measuring the impact of single projects; those targeted at very specific organizational issues. But there may be times when you want to measure the impact of a broader solution.
Back in 2016, we worked with Edrington to deploy our content library and won Gold in the ‘External Learning Solution of the Year’ category at the Learning Awards. Our awards submission was largely based on our ROI calculation, which adopted a variation of Robert Brinkerhoff’s Success Case Method.
In essence, we ran a survey across Edrington asking how much time colleagues felt they had saved by using our performance support resources. We then calculated an average, multipled that by the average hourly salary, and came up with a projected cost saving for the business as a whole. Respondents who had saved the most time, or the least, were interviewed to provide context to the results.
🚀 Hurling cash into orbit
Calculating return on investment can sound complicated, and those of us who work in HR and L&D rarely have a financial background. My own degree is in journalism and creative writing (Shameless plug: Buy my book, Centauri’s Shadow, available now on Amazon and rated 4.9 out of 5 stars!).
But it also isn’t as hard as it seems.
It’s true that JFK’s goal was not a financial one. But the money spent on the space race wasn’t spent in orbit, it was spent at home. It created jobs for 400,000 Americans, and led to the invention of freeze-dried foods, cooling suits, kidney dialysis machines, insulation for oil pipelines, building materials, physical therapy machines, flame-resistant textiles for firefighters, water purifiers and integrated circuits. NASA has a team responsible for tracking the commercial applications of the technology it has helped develop, and the ROI on the Apollo program.
These innovations were not planned upfront: they are a consequence of the space program, not an objective. And this isn’t an argument for hurling cash at any ambitious goal without careful planning.
But calculating ROI isn’t as out of this world as it might initially seem.
Have other ideas for calculating the ROI of learning? We’d love to hear them! Want to discuss how the Mind Tools Custom team can help you see an ROI on your projects? Get in touch by emailing custom@mindtools.com or reply to this newsletter from your inbox.
🎧 On the podcast
Long-term listeners and readers will know that Ross D and I are obsessed with productivity hacks. ‘Getting Things Done’ is a Ross G favourite. ‘Deep Work’ is Ross D’s go-to.
So Marc Zao-Sanders’ new book, Timeboxing, is catnip to us.
In this week’s episode of The Mind Tools Podcast, Ross G shameless invited Marc for a 1:1 coaching session on how to better use his calendar, structure his days, and focus in ‘one thing at a time’.
Check out the episode below. 👇
You can subscribe to the podcast on iTunes, Spotify or the podcast page of our website. Want to share your thoughts? Get in touch @RossDickieMT, @RossGarnerMT or #MindToolsPodcast
📖 Deep dive
I’ve been watching 3 Body Problem on Netflix, which is absolutely dreadful. And not, as I first believed, a sequel to Benjamin Bloom’s 2 Sigma Problem.
This joke is so unbelievably niche that it can only be improved with a detailed explanation.
In 1984, educational psychologist Benjamin Bloom (he of ‘Bloom’s Taxonomy’) found that the final achievement measures for students who received tutoring was about two standard deviations above those in a conventional classroom environment (hence the ‘2 sigma problem’).
More simply:
‘The average tutored student was above 98% of the students in the control class.’
This result won’t come as a surprise to most readers. We would expect individual tutoring to be more beneficial than a typical classroom environment. But herein lies the problem: How can we scale tutoring so that an individual educator can provide the same level of instruction to an entire class?
With the help of his graduate students, Bloom embarked on a search to find those variables that could be adjusted to deliver similar results as one-on-one tutoring but on a larger scale.
The most impactful factors were reinforcement of key learning points, corrective feedback when students make mistakes, explanations of concepts, active participation in their learning, the time students spend on learning tasks, and the students' own study skills.
While none were as effective as tutoring alone, the development of adaptive learning courses and AI tutors in the decades since the paper's publication have been making strides toward a scalable solution.
Here at Mind Tools, we've successfully integrated these insights into our learning designs to enhance educational outcomes:
Personalize learning content: Our Custom team offers learners different routes through digital experiences, allowing them to 'choose their own adventure' based on their role, seniority, or prior experience. This might include branching scenarios and scenario-based learning that mirror real-world challenges.
Provide tailored corrective feedback: Our AI Conversations tool allows learners to engage in simulated dialogues with a chatbot, evaluating their responses against a bespoke framework. This approach offers all the benefits of role-play without the discomfort, providing precise feedback in five crucial areas.
Encourage active participation: Our Skill Bites courses provide weekly insights into key management techniques and prompt learners to commit to personalized, context-specific actions. Learners are then encouraged to reflect on their experiences the following week, solidifying the learning through practical application.
How do you incorporate Bloom’s theories into your learning designs? We’re eager to learn from your experiences. Hit reply to this email and let us know!
Bloom, B. S. (1984). The 2 Sigma Problem: The Search for Methods of Group Instruction as Effective as One-to-One Tutoring. Educational Researcher, 13, 4-16.
👹 Missing links
🤯 Am I falling out of love with Freakonomics?
Y’all know I’m a sucker for Freakonomics, the bestselling book from economist Stephen Levitt and journalist Stephen Dubner. First published in 2005, it evolved into a multimedia empire with the the duo promising to uncover ‘the hidden side of everything’ through trendy statistical analysis and counterintuitive solutions. It’s an attractive concept that spawned enough copycats to fill an airport bookshop, but it sparked widespread arguments within academia and, last month, Levitt decided it was time to get out. Basing his decision on the near-lack-of-interest in his academic papers when compared to the millions of listeners to Freakonomics Radio, Levitt is taking his ideas elsewhere. This article, from The Economist, does a good job of summarizing his academic legacy.
🤖 A whistlestop tour of the thinkers who influenced generative AI
Many readers will be familiar with Donald Clark: speaker, blogger, advisor and researcher. In this episode of Great Minds on Learning with John Helmer, recorded live at Online Educa in Berlin, Donald takes listeners through the contributions of more than 20 researchers whose insights have shaped the genAI tools we’re using today. It’s a fast-paced and snappy introduction to how we got where we are now.
💗 Timeboxing for your mental health
As well as appearing on The Mind Tools L&D Podcast, our pal Marc Zao-Sanders also recently did us the courtesy of recommending our L&D Dispatch newsletter. That’s led to a big spike in readers so, if you’re one of them, welcome to our corner of the interwebs! It now seems only right that we thank Marc for the introduction, and return the favor with a plug for his latest newsletter: ‘Small steps for mental health’. It takes you through how timeboxing can help you reduce stress, sleep better, and focus on ‘one thing at a time’. Enjoy!
👋 And finally…
While clearing out our Dropbox this week, I found some archive training videos we produced.
Kidding, kidding… they’re not ours. But, I wish they were…
Tiktok failed to load.Enable 3rd party cookies or use another browser
👍 Thanks!
Thanks for reading The L&D Dispatch from Mind Tools! If you’d like to speak to us, work with us, or make a suggestion, you can email custom@mindtools.com.
Or just hit reply to this email!
Hey here’s a thing! If you’ve reached all the way to the end of this newsletter, then you must really love it!
Why not share that love by hitting the button below, or just forward it to a friend?