Schools’ use of ed-tech has increased exponentially in the past year. It has been a remarkable period of change. Schools are now more knowledgeable about how they can use ed-tech and, most importantly, what works for their local context.
Schools have learned that not all approaches and tools work for them and that what works well for one school may not be best for another.
Just ahead of the first lockdown, nine leading ed-tech companies founded the EdTech Evidence Group (EEG). Our goal was to encourage ed-tech companies to provide clear evidence about the impact of their products and services in the classroom. A year on, this agenda is even more important.
Schools have accelerated their understanding and confidence in using ed-tech to support learning, but without solid evidence there is still huge potential for money and time to be wasted on ineffective solutions.
Demand good ed-tech evidence
The combined experience of the EEG is that schools are not seeking and scrutinising evidence as much as we would like when they procure ed-tech; in fact we are surprised that a lot of schools rarely ask for evidence.
Instead, schools regularly rely on word of mouth and references from other schools that they trust when considering ed-tech purchases. This is a useful starting point but there are risks. Schools are all different and what works in one school may not be right in another.
We are also concerned that schools are often persuaded by high numbers of customers – on its own a poor indicator of an effective product.
When schools do discuss evidence, the focus is generally on outcomes or academic progress. Matt Koster-Marcon, founder of Learning Ladders and a member of the EEG, says schools need to ask how tools will specifically improve learning in their context. Also vital he says is asking about the nitty-gritty around service-level agreements to avoid being burnt by “some companies over-promising and under-delivering”.
What does good evidence look like?
When it comes to finding evidence a good place to start is a company’s website. Research will usually be featured clearly – if you can’t find it then get in touch and request it. A healthy mixture of research is a good sign that an ed-tech company is taking evidence seriously, and that you will be able to understand whether their offering is right for the problem you are trying to solve. Look for rigorous, thorough evidence and do not be afraid to challenge what you are given.
Independent studies undertaken by research organisations or universities mean the research has to meet particular standards. Often cited as the gold standard is the random control trial, which removes the risk of bias, however these are costly so can be out of reach for many ed-tech companies.
Other less costly approaches such as school surveys are also useful but be sure to check the sample size. If there are only a small number of respondents then the findings cannot be extrapolated.
Case studies can be undervalued, but if they are detailed they can be very useful. Murray Morrison, founder of Tassomai and a member of the EEG, explained: “Case studies can present vitally useful information to a school. Their value is to show to a school the ‘how’ – how did a school like mine use this tool to achieve the kind of changes that we seek to achieve ourselves?” Where possible, seek out case studies from schools whose context is similar to your own.
Look out for solid, objective research which proves the efficacy of a product. A healthy mixture of quantitative and qualitative research and clear explanations about what it shows should be available. Stay focused on the problem you are trying to solve and whether the research suggests it will help to solve it.
The EEG is also trying to encourage ed-tech companies to have a dedicated webpage where they share the evidence they have about their products.
Getting the most from trials and pilots
Ed-tech research is notoriously difficult because of the huge number of variables that exist, even in just one classroom. This means it can be hard to understand whether an ed-tech tool will be right in your particular context, or meet your specific needs.
Many companies offer free trials and are open to running pilots with schools. I would recommend you use these to help understand whether a solution is right. Free trials are usually relatively short – four to six weeks is typical. To get the most from a free trial ensure you plan ahead.
- Get advice from the company and tap into their expertise.
- Recruit a group of teachers to try the product and agree together what you want to test.
- Review the trial regularly as a group to gather as much information as possible.
- Get feedback from students and any other stakeholders that have been involved.
- Try to gather quantitative and qualitative data if possible.
Generally, pilots will be for a longer period of time, anything from a term to a whole school year. The ed-tech solution is usually purchased, possibly for a small number of classes, which can reduce the cost in some cases. This provides more insight and a richer set of data and, once completed, a school could then roll-out more widely.
Getting involved as a test bed school
Many ed-tech providers want to provide more evidence, but can face logistical difficulties in recruiting schools who might want to test a product for a longer period of time. It is worth discussing this with an ed-tech company to understand what would be involved, as there can be benefits in acting as a test bed.
For example, my company – Sparx – has developed close links with our local schools in Exeter where we have been able to undertake detailed, long-term research. They have been able to shape Sparx to meet their needs, have had access to extra technical support, and received discounted fees for access.
Looking ahead
School leaders now have a much better understanding of the ed-tech that has “made the cut” during the past year, and the tools that have languished unused.
We now have a teaching workforce who are fast becoming ed-tech experts with more knowledge and experience. It is an exciting time.
However, it is crucial that ed-tech doesn’t lose sight of its core purpose: to support teachers and students and enable them to work and learn better and more efficiently.
Without evidence about the efficacy of our products this simply is not possible. So I urge school leaders to challenge ed-tech companies to share their evidence.
- Dan Sandhu is CEO of Sparx and a co-founder of the EdTech Evidence Group For more details, including the group’s evidence checklist and guide “What does good evidence look like?”, visit www.edtechevidencegroup.com