Insights interview: why investing in learning science for your edtech makes good business sense

What’s the purpose of this interview and who’s it for?

Investing in learning science can sound like an indulgence to an edtech company that’s short on cash and needs to move fast. But, in the broadest sense, learning science is about using evidence to make smarter decisions. And, that’s the process of reducing risk and increasing likelihood of success. So, I hope to convince you that investing in learning science for your edtech makes good business sense all around—for your customers (in experience and educational outcomes) and your business (in investments and market success).

At the risk of offending all edtech entrepreneurs reading this, in talking with hundreds of startups through my career (and assessing or advising nearly 150), I routinely find leaders who are super-strong in one area of using evidence, but weak in others. Specifically, edtech companies are often driven by innovation and expertise in the business model, teaching/pedagogy, user experience, or teacher/instructor experience. And, in that area of bench strength, the company uses evidence rigorously and instinctively to make smart decisions—for example, balanced market analysis to drive innovation in a business model, or collaborative teacher workshopping to guide an intuitive user experience. However, those same companies are often weaker (that is, less mature) in using evidence in other areas that are also necessary to design, evolve, and grow edtech successfully in the market.

This is where I think investing in learning science—and, in particular, using evidence—isn’t an indulgence, it’s fundamental to building a solid edtech business. And, the earlier you incorporate it, the healthier your awareness of risks and challenges and the smarter your decision making and investments.

We are also facing some acute challenges in education that learning science can help entrepreneurial edtech companies to tackle. In particular, helping traditionally marginalized students to be more successful (educational equity), and at-risk K12 students to better re-integrate through and beyond COVID-19.

Why Dr. David Porcaro?

With these ideas in mind, I wanted to pick the brain of Dr. David Porcaro, Director of Learning Engineering at the Chan Zuckerberg Initiative. David is unusual because he’s a card-carrying Learning Scientist who’s also led the application of learning science in edtech companies of various stages, shapes, and sizes. So, he has deep and practical experience about the educational, strategic, operational, and commercial benefits—and also the road bumps—of using learning science in edtech design that I think many edtech entrepreneurs can benefit from.

The interview

Hi David—I know you’ve just climbed Mount Whitney, so I’m sorry to bring you back to terra firma! As you know, I help edtech companies around the world to grow their businesses. Most leaders are wrestling with priorities of cost, speed, and impact. In my experience, learning science in its broadest sense—using evidence to reduce risk and make smarter investment decisions—can really help them. In that context, I want to tap your practical experience to answer these questions:

1. Why, where, when, and how should edtech leaders invest in learning science?
2. How can edtech leaders use evidence to reduce their risk, evolve their edtech, and mature their business?
3. How can learning science enable edtech leaders to tackle educational equity?
4. How can learning science guide edtech leaders who are exploring ways to help teachers and students to re-integrate through and post COVID-19?

Sure, thanks for inviting me to share some of the experiences and insights I’ve gathered from working with so many amazing people globally in education. And I think you’ve nailed the four things I spend most of my time thinking about these days.

1. Why, where, when, and how to invest in learning science

Let’s start with the business “why?” Why does investing in learning science for an edtech company make good business sense?

Well, I’ll start by saying because it’s the right thing to do! We wouldn’t ask a chemist working on a COVID-19 vaccine if using science was worth the investment. Sometimes in education, we rely on our intuition or personal experiences too much. 

The most important lesson I’ve learned as a learning scientist is that how I learned as a student may not have been optimal, may not reflect the diversity of experiences of other students, or may not work in all contexts. So, don’t rely on your own immediate experiences.

Learning science research helps us to create tools and resources that do what we really hope they will-—improve students’ ability to learn. And in an increasingly noisy world of edtech, products that really help will be those that teachers, parents and students adopt. 

Policymaking and education funding are also increasingly guided by evidence. In the US for example, federal funding for edtech often requires evidence of impact. That demand is only going to increase with time. So I suppose that soon the answer to your question will be “because your market requires it.”

Finally, from a business perspective, building products using evidence will save you money, get you faster to product-market fit, and ultimately get you happier paying customers. You can find that out through long and expensive experiments—product launches and updates—or faster and more efficiently by applying learning engineering through your product development.

Let’s move to the opportunity “where?” Where can edtech companies most benefit from applying learning science?

Learning science should really be applied to every aspect of product development. It should play a role in the development of products and features that directly impact learning, and also in the development of professional support to enable educators to make effective use of those tools.

Let me provide some more color. There are two parts to applying learning science. The first is in the development of your ideas, designs, and products. As you make design decisions—from your key value proposition to how you write effective answer feedback—you will make lots of decisions, some of which will really impact learning. Have you thought through the logic of those decisions? Will the solutions and professional development you propose lead to the academic and socio-emotional outcomes you’re aiming for? What unintended consequences could there be and where are your riskiest assumptions—where if you got it wrong the negative impacts on learning would be most significant? And where do you feel the least confident about your assumptions? 

As you identify weak spots in your logic, you can begin to fill them in with evidence. Firstly, focus on your own users and how they engage with your products. This could be through observation or User Experience Research (UXR), or perhaps platform data and learning analytics. And also search published research—ideally, high-quality syntheses of studies across multiple settings. These studies won’t transfer perfectly (because of likely differences in context), so you’ll need to think carefully about what applies to your context.

This is where the second part of learning science comes in—data collection. You will have a product hypothesis by this stage, and a good theory of what you think will lead to learning outcomes. And you will likely get some of it wrong. So you also need to set yourself up to collect formative data (observational and experimental) that you can use to quickly and iteratively refine your hypotheses and improve your product. As you involve more users, and the stakes get higher, your methods of collecting and analyzing data will need to get more rigorous so you can start to develop a solid understanding of what works, for which learners, and in what contexts.

[See also Six key principles for learning experience design, How to design edtech for impact, Unpacking the Black Box of Efficacy, and Why and how to invest in edtech data for your growth.]

Given CZI’s mission to help individual learners to achieve their potential, where can edtech use learning science to best help individual learners?

Our mission centers on understanding how learners vary, and how serving that variability can be built into edtech and teaching practice. For instance, one of our partners, Digital Promise, has built a tool to help organizations understand the variability in K12 learners to guide product development. Through years of research in learning science, we’ve learned that when you build for the average student you leave many in the margins. This is especially true for students eligible for special education services or with non-dominant languages. Additionally, we know that strategies that work well for some learners can often have the opposite effect on others—for instance novice versus expert learners—what’s called the expertise reversal effect.

As you develop a clearer picture of the variability of learners and their contexts, you can start to develop personas you can design for and test against. For example, how do your user personas differ in their prior knowledge, motivations, mindsets, home lives, access to technology, and social and emotional development? Without taking these into consideration, we risk only helping those who already have the supports to be successful—in other words, giving more to those who have, and leaving behind those who don’t.

For the teacher, where can edtech use learning science to turn data into insights to help individual learners?

I believe it’s really important for teachers to build strong relationships with their students. This is the key to motivation which is ultimately the key to learning. If students don’t feel they belong in a learning environment, they’ll spend precious cognitive resources focused on their lack of belonging—not on learning. So it’s essential for teachers to use whatever tools they can to better connect with students and learn about their interests, strengths, super powers, and goals, as well as the supports they need to meet the objectives of the course, school, or product. 

To do this, teachers need reliable and consistent formative assessment. How did changing the way I introduced the lesson impact David’s ability to seek help? With edtech tools, teachers can run daily, weekly, or even longer longitudinal “experiments,” and continuously improve their practice. The strongest teachers are essentially learning scientists, creating hypotheses, designing interventions, running experiments, and iterating. Think about how your edtech can facilitate that.

[See also How to design more actionable edtech dashboards: design principles and examples, and Drowning in data, or swimming in insights? Keynote at ELT Young Learners’ summit.]

Let’s move on to the critical question of “when?” With limited budget and time, when would you advise an edtech leader to invest first in learning science and why?

I’d start by keeping it simple: What evidence does your team bring to key decisions? Are your decisions based on intuition, your leadership team’s lived experience, or market trends? Then, target your evidence collection on where you have gaps. For example: 

• Are you weak in collecting and interpreting the needs and behaviors of your users? Build up your User Experience Capabilities. 
• Are you struggling to sharply define your strategy? Strengthen your market research with a learning science advisor. 
• Are you lacking insight into how students are using your product and learning with it? Build up your learning analytics and data-science team. 

You can start with advisors, or contractors. Then, bring on a learning engineer (or learning designer) who can bridge these gaps as you build up your capabilities. Eventually, you may build a full team that would comprise UX researchers, data scientists, measurement and evaluation specialists, and learning engineers fully embedded in your product development.

If they can afford to hire, what advice would you give on recruiting their first learning science talent? What qualities and skills should they look for? Where should they look?

For your first hire, you need someone who is a translator and boundary spanner. This person should know just enough about human-centered design, product development, learning science, evaluation and research, content development and business and strategy to be able to turn research insights about learning into actions that a variety of people across the organization can use.

The best learning engineers I’ve hired are excellent listeners and can look at complex, dynamic problems and come up with very actionable support and guidance—all on top of a solid understanding of learning and psychology. They also need to appreciate commercial considerations—what learning solutions can be scaled affordably, and which are practical for a customer to implement. They also need to believe that design is a team sport.

In the US, the Stanford’s Learning, Design, and Technology program, Carnegie Mellon’s Masters of Educational Technology and Applied Learning Science program, or Boston College’s new Learning Engineering program are all training learning engineers. So, I’d start there.

Now, let’s look at the operational “how?” What organizational or process lessons have you learned helping product teams to readily apply learning science?

Start where you are. Are you ready to add learning impact goals to your strategic KPIs? Are you ready to evolve your data collection to capture one or two measures of learning, motivation, or mindset? Over time you can progressively close gaps between where you are today on data collection and where you want to be.

Also, connect with others. For instance, in the US, you may be eligible for a grant from the Institute of Education Science (part of the Department of Education) or National Science Foundation to engage a learning scientist to research your product. Also, consider building an advisory team with specialities in learning engineering or design to give you on-going critical advice.

What resources would you point edtech leaders to for practical advice?

There are a number of organizations helping to codify learning engineering. For instance, we’ve worked with the ISTE (International Society for Technology in Education) to create resources for better explaining the science of learning to teachers. We’ve worked with the EdTech Evidence Exchange on their EdTech Genome project to create a framework for describing contexts to help research into how context shapes education results. We’ve supported organizations interested in learning engineering and evidence centered design, including ASU-GSV, the International Society for Design and Development in Education, and the Industry Consortium on Learning Engineering (ICICLE). There are a lot of great examples of people sharing practical guidance in these initiatives or on listservs of how this is done across different kinds of organizations.

[See also those recommended by Professor Chris Dede in How to engineer edtech for learning and adoption at scale.]

2. How to use evidence to reduce risk, evolve your edtech, and mature your business

I’ve found that edtech companies are often powered by expertise in the business model, teaching/pedagogy, user experience, or teacher/instructor experience, but rarely all of these. That creates risk as a company invests to develop and mature its offering. How do you go about helping an organization to understand where it is weak?

I’ve come to find that companies develop these skills at different rates. We’ve been working on building up a maturity model, not unlike the design maturity models out there that say “if you do this, you’re at maturity stage X, and we recommend you build up these muscles.” Moving through these stages takes time (some of these models suggest it takes an organization several years to evolve to the next stage of design maturity). Each organization is going to be a bit different, but as you start to ask “how is my product impacting learning” you’ll soon find where your biggest holes are.

What are some examples of affordable, swift, and practical steps the company can take to shore up areas where they are weaker?

Most of the best learning engineering comes not from digging up the right answers, but asking the right questions. And those are free. Ask more questions like:

• “What do I want a learner to decide and do after using my product?” 
• “What do I expect to see in a setting where people are using my product?” 
• “What needs to be true for the optimal learning context?” 
• “How does this decision impact the learner and learning?” 
• “How do different types of learners experience my product?” 
• “How do I know if learners have been successful?” 
• “What could get in the way of learners having success with my product?”

I ask these questions of product teams almost every day. And then I push people to write down the evidence they have for each answer. Did it come from asking users? From a published article? From A/B tests? Or, from years of practical experience? Having a way to document why you made the decisions you did, and what type of evidence led to those decisions is a simple but powerful way to start.

3. How learning science can enable edtech to tackle educational equity

To hit the largest sector of the market, edtech companies often aim for the “middle of the market.” But, as a result, their technology may by design marginalize learners already in the margins. How can K12 edtech companies engineer for learning equity for all students?

In education you have an obligation to serve all students. Once you design a product with users to meet their learning goals, the next step is to see how your product actually impacts different types of students. Surprisingly too few companies research this. Is your product only available to those who have wifi access at home, or can afford expensive hardware? Does the user have to read and write English (or another language) fluently to engage with your product? Does the user have to be visually, orally, or mobility-abled to complete the learning activities? Do your content, color scheme, images, metaphors, or algorithms unintentionally make students feel marginalized or like they don’t belong? If that’s the case, they’re not going to be able to or want to use your product, and they won’t learn. Unless you measure and track these things, you won’t know how to address them. So start by asking these questions and do what you have to to get the answers—even when they’re hard to come by.

Now, let’s jump ahead. So, you’ve discovered your product isn’t used by low–income or underrepresented students. How do you address that? Well, this is where you’re going to have to cede some of your control of the product to your users. There are many great participatory design practices, including Equity by Design, community-driven research, design-based implementation research, and other tools for capturing the voice of all of your customers, including students, parents, and community members, into your product decisions. There’s also a really great toolkit for designing more equitable AI in edtech.

How do you square this with a company’s business need to serve many schools and teachers with diverse students?

In the US, as well as many other countries, more than 50% of students are in underrepresented racial or ethnic groups. Creating diverse, student-centered products is not only a legal requirement, it’s good business. The more you understand learner variability, and the more you design to flexibly account for that, the better your product-market fit, and the larger your target market. That may mean simplifying your design, or providing more customizability on the user end, or it may mean building user communities to allow your users to share and learn from each other. Each scenario is different, but it’s not impossible. You just have to make it a priority.

4. How learning science can power edtech to help teachers and students to re-integrate through and post COVID-19

Schools, teachers, parents, and students face unprecedented challenges re-integrating through and after COVID-19. There have been many articles highlighting that where there were previously cracks to be closed with students at risk, now there are chasms to be bridged. Where do you see as the most promising opportunities for edtech companies to exploit learning science to help teachers, schools, and parents re-integrate K12 students?

There is no doubt about it, this is a really hard time. Teachers and schools are suddenly working with new, untested tools and methods. Families are stressed, scared, and suffering. Students are falling behind grade level. Before we rush to try to speed up math skills that may have been lost or try to cover missed standards, we need to build an inclusive safe, nurturing environment that is trauma informed, and addresses the whole student. If we don’t attend first to issues like health, safety, mental wellbeing, nutrition, and connection, no digital tool is going to work. At CZI we’ve invested heavily in helping schools to manage the transition back to school, and we continue to stress the socio-emotional aspects of re-integration into school. It’s the schools, programs and interventions that are focusing first on students’ and teachers’ sense of safety, connection, and hope for the future that I am most excited about.

David, this has been awesome. I share many of the same convictions about the good well-designed edtech can deliver, challenges organizations face, and practical ways learning science can help. Your practical examples will be very helpful to many. Thank you so much for making time for this interview.

Thank you so much for having me. It’s always a pleasure discussing the future of education with you, Adam!

Takeaways

David provides many valuable insights and lots of links to super-valuable resources. The biggest takeaways I’d draw your attention to about why investing in learning science for your edtech makes good business sense are these:

1. Build with learning science to save time and money. Building edtech with evidence will save you money (less design churn), get you faster to product-market-fit (using customer insights), and ultimately get more satisfied customers (they shaped the solution).
2. Build with learning science because it’s your moral obligation. Building edtech with learning science so your product delivers better learner outcomes is what you owe your customers, and gets you ahead of growing governmental and market demands.
3. If your company is truly committed to learners, put learning goals in your business KPIs. Learning impact goals are the benefits your edtech delivers to your customers, so put them front and center in your business performance goals (p.s. historically, the stickiest edtech delivers great outcomes).
4. To mature your business, identify where you have gaps using evidence for decision making. Identify which teams are strong or weak using data for decision making and build a roadmap to fill the gaps to reduce your risk, invest smarter, and increase your likelihood of success.
5. Strategically tackle delivering educational equity (it’s also a growth requirement). “When you build for the average student, you leave many in the margins.” The more you design your edtech to flexibly serve learners’ different needs, the more you will help those marginalized and the greater your growth potential in diverse markets.
6. Build tools that help teachers to build stronger relationships with their students. Build tools that help teachers to gain richer insights into their students interests, strengths, super powers, and goals, and give them formative insights to help them experiment and improve.

I had the privilege of building an incredible learning science team at Macmillan Learning. We applied these techniques full bore to the development of Achieve which won Digital Promise‘s “research-based design” award earlier this year. The full report describes the techniques we developed.

If you want a fast and incisive assessment of gaps in your current approach (of leveraging learning science and using data and evidence) and practical recommendations for how to close these, please contact us today with the form provided below.