Using data-driven approaches at school? Avoid these mistakes

The idea: Here are 3 common mistakes to avoid when adopting data-driven approaches in your schools: #1 Lacking a clear purpose for collecting data, #2 Failing to tackle data aversion/ fearing mindsets in the community, #3 Drawing conclusions without analyzing the reliability of the data 

For any School that wants to be innovative and improve quickly, data-informed approaches/ improvement science approaches should be central to their functioning. Read more about adopting an Improvement Science Approach in schools in this book: Learning to Improve: How Americas Schools Can Get Better At Getting Better.

In a nutshell, it is a continuous cycle of improvement that looks like this: 

Plan: Plan what and how you want to improve. 

Plan what you need to collect as evidence of improvement→ 

Do: Implement your plan and collect your evidence/ data → 

Study: Analyze the data and use it to make decisions on how to do it better → 

Act: Put your decisions into action →

Next, begin the Plan-Do-Study-Act Cycle again

In the last 5 years, I have had the privilege of working with my organizations to make this shift happen. While there is so much I have learned along this road, there are 3 common mistakes I have observed. Here they are:

#1 Purposeless: Collect it all!

Every piece of data collected needs to be intentionally collected with a purpose for use. Collecting data 'just in case it is useful later’ is always a bad idea. Planning for and collecting data takes time, and a purposeless approach wastes valuable time and energy of your team. It can inundate them with meaningless work. Also worth mentioning are the more deep rooted problems this results in: Not understanding ‘why’ you are doing something leads to an unfeeling, unthinking approach to a process that should have critical thought at its core. It also cultivates an attitude of indifference or even aversion towards the use of data in learning and improvement.

How to combat this:

• No one should ever say, 'just collect this data and then we’ll think about how to use it later'. Everyone should know why they are collecting the data and what they will using that data to improve, before they begin collecting and studying the data.
• If you are just starting up, start slow. Pick a few pieces of data that you would like to collect (with a purpose, of course) and carry out a cycle of improvement with that data first. Then slowly build up from there towards collecting more comprehensive data, being thoughtful about the purpose and feasibility. 
• Leaders have to be extra mindful here to be purpose-driven, since these processes are usually implemented at the organisational level and they are often driving them.

#2: Data Aversion: I’m not a data person

After spending 6 years working with engineers, I was nothing short of shocked to see how much my education community (apart from the few math, science and tech teachers) disliked and/or feared data! The good news is, all of this data aversion or fear is easily countered by talking about the mindset and building data literacy in the team. However, if not addressed, this fear can put a giant spoke in the wheels of your data-driven practices. It interferes with one's ability to build data literacy and critical thinking in general.  

How to combat this: You will need to do two things to combat this - change beliefs and build competence.

• Teachers need to believe that everyone is capable of using data to improve. Fixed mindsets about ‘not being a numbers person’ can be detrimental to one's ability to learn even under the correct circumstances. As a leader you need to believe this too. Helping teachers recognize this mindset is an important first step.
• However, you can’t combat a fear of data by only changing one’s belief, you have to support it by building their competence with data too. At the start, scaffold the process of data analysis to build data literacy in the team. Soon they start to see themselves use data successfully and see the value it adds to their learning process. Once this happens, you have set in motion the wheels of self-perpetuating improvement. Side note: If you haven't discovered the magic of datastudio yet, here is a great course for educators to get you started.

#3 Not Enough Doubt:  Can you trust this data?

Another critical mistake I have noticed is that people are too eager to use data to draw conclusions. They do this without examining the reliability of the data or questioning the fidelity of the data collection process. Quite simply, if the data is not trustworthy or the limitations of the data are not understood properly, it will lead to poor decisions with potentially bad consequences. 

How to  combat this: Your process of analyzing data should always comprise of these two crucial steps: 

• Examining how trustworthy your data is: Was the data collection process done with fidelity? Are there any possibilities of the data being wrong? Is the data comprehensive enough for the decisions you are using it for or are there some limitations?
• Using data to inform your decisions only after you have evaluated point 1: Critical analysis of your data to draw conclusions and inform decisions should only happen after you have evaluated the reliability of your data. If your data is not 100% trustworthy (as it will be in many situations in schools) you may consider using it with caution. An effective way of doing this is by using it along  with other sources of data including qualitative ones.

Remember, using data-driven approaches for improving is a continuous process. As you use it more, you learn more and get better at planning, collecting and analyzing data. 

If you or your team are trying to incorporate more data into your decision making, I hope this article helps you avoid these common pitfalls.

Please note: When complex ideas are simplified into basic models to aid thinking, we lose some nuance. It is important to remember this limitation :)