How to design more impactful edtech using levels of evidence

Efficacy research will help you to design a more competitive product, faster

Investing in efficacy research can sound like an indulgence for an edtech startup that’s moving quickly, staying lean, and needing to build, evolve, and grow its product. And often companies put it off until “the product is done.” But from experience taking many major edtech products to market, I’ve found that smart investments in efficacy research will help you to design edtech that’s more impactful, with greater competitive advantage, and faster, and levels of evidence provide a powerful organizing framework.

Four time-tested reasons to invest in efficacy research

I’ve blogged extensively about why investing in learning science makes good business sense, the power of using outcomes and efficacy as your North Star, and the value of fast and reliable insights to guide your investments and formatively refine your product. In essence, these boil down to four core reasons to invest in efficacy research as part of your product development and evolution:

1. Investing in efficacy search will save you time and money. This may sound counterintuitive. But, gathering evidence and using it to inform your decision-making puts your customers at the center of your process (and keeps them there), results in less churn during design, and will help you to achieve product-market-fit faster (saving you on expensive re-engineering and lost momentum downstream).
2. Selling better customer outcomes (with evidence) is more compelling than selling features (and benefits). Outcomes are what your customers and users ultimately care about. They don’t want to talk about widgets and (most) are tired of product-centered discussions. Using improved outcomes as your value propositions keeps you laser-focused on your customers and enabling their success. 
3. Honesty is fundamental to building trust with your customers and to retaining them. If you stretch the truth or exaggerate claims of what your product does early on (and many edtech startups fall into this trap), it will catch up with you and nobody will believe you when you scale and start to compete head-on with more entrenched competitors.
4. Learners have one chance with their education. If you’re in the business of education, hopefully you want to help learners and teachers. So, you owe it to your customers and users to build products that will help them to achieve their best and you can’t do that without embedding efficacy research into your development and validation.

Where to start—the business rationale

From experience, busy executives and investors will reasonably challenge the commercial benefits of any investment. So, I like to start at the end—namely, bold, competitive (and honest) marketing claims. Here’s a pyramid of increasingly competitive claims, starting at the bottom with claims such as your product is designed using learning science and educational research (in other words, it’s plausible that it should be effective), to users use your product a lot, like it, and perceive that it’s valuable (that suggests it’s easy to adopt and has good usability), through to demonstrating learners actually achieve better results because of your product (i.e. a cause and effect). This helps you to frame the questions “How bold is our ambition?”, and “What do we need to be competitive?”.

The how—what research to invest in and when

Now you’ve got the attention of your execs or investors, you can pull back the curtain and reveal what practical, affordable, and specific research needs to be invested in as part of product development. [In this blog, I won’t get into costs and the pros and cons of recruiting the right talent vs. outsourcing these tasks.] Boiled down from years of building product and research teams, refining processes, and pitching the business benefits, I’ve distilled this into the following crisp tasks—from investing in user and learning research to identify customer needs and solutions that leverage “what works” from learning science, through to combining surveys and product data to start to explore if there are relationships between implementation (how a teacher or instructor chooses to use the product), student use, and outcomes (such as improved engagement, perceptions, and performance), right through to Randomized Control Trials or Longitudinal studies where studies are performed over enough contexts and users that key factors can be controlled to really measure the role of the product itself.

Building a roadmap for how to design more impactful edtech using levels of evidence

So, what efficacy research should you plan, and when, to design more impactful edtech using levels of evidence? Pulling this together, you can now see research to invest in on the left and claims this will enable on the right. Of course, investing in research doesn’t guarantee that the results of your product are what you hoped they would be. And, that’s the pivot of embedding research into your product development—because you get unvarnished insights at each stage which enable you to efficiently improve the product during development, refine faster, and ultimately deliver an experience that drives better outcomes and claims.

A few important notes

This framework is designed for edtech startups—to help you to understand how to design more impactful edtech using levels of evidence, make a case for investment, and plan your yearly approach. With that audience in mind, here are two important things to note:

1. Iterative. This schematic of levels may suggest that the process is linear (climbing to the summit) and that product development is sequential and waterfall. In practice, edtech development needs to be iterative and continuously evolving. A pragmatic and experienced efficacy researcher will help you to embed these research methods into formative early product development, shadowing your product in market as it scales, and on-going discovery activities.

2. Formative. A common mistake edtech startups make is to delay efficacy research until the product is “ready” or they “need” evidence to win specific business. That’s too late. It will take you several years to build that evidence. And, once you have results, you will find you need to re-engineer your product to make it impactful. In the meantime, you will have been selling an educational product with no understanding of whether it’s effective (ask yourself this: would it be moral to sell a drug and then run trials afterwards to find out if it works?). Invest in research early, and you’ll refine your product faster, at lower cost, and more efficiently, and you can launch to market with (directional) evidence of benefits.

Thanks and other resources

There are lots of frameworks for efficacy research, such as the Standards of evidence from Nesta, the ESSA (Every Student Everywhere) tiers of evidence, and What Works Clearinghouse (applying ESSA Tiers of Evidence), to more holistic rubrics for evaluating course design (from Quality Matters). Most of these are tailored for researchers, or bodies benchmarking the evidence supporting a particular product. When I first developed this particular version (with @Claire Masson back in 2005!), our goal to was to persuade edtech executives on the value of investing in efficacy research. And, since then I’ve continued to evolve the methods and phrasing to make it practical and approachable for edtech startups. If you’re an edtech startup, I hope it’s a helpful introduction for how to design more impactful edtech using levels of evidence.

Need help? Have feedback?

Here’s a downloadable PDF of this deck. If you have bold ambitions for your edtech and want help with designing a practical efficacy strategy, or you’re a mature education company exploring how to improve the competitive advantages and positioning of your services, we’d love to help. Please contact us today with the form provided below. We also welcome your feedback or questions—just use the form and title “Your challenge” with the word “Feedback.”